Re: auotscaling tlog+pull and replicationFactor

2020-04-02 Thread Sandeep Dharembra 
I am not sure whether you have added triggers for nodelost and nodeadd
events

Action is taken based on triggers which have the execute plan. You can move
create replicas and also delete node in these triggers.


One more thing, since you are using solr 7.7 if you add move replica action
to the nodeadd trigger it will be fine but if you want to use addreplica on
nodeadd event, this version of solr supports only addition of nrt type. If
you want to add pull the support was added to specify replica type in solr
8.4

On Thu, Apr 2, 2020, 7:51 PM Richard Goodman 
wrote:

> Hi there,
>
> I'm currently using solr v7.7.2 *(I applied the patch SOLR-13674
>  to my build to preserve
> replica types for triggers)* and trying to set up a cluster with the
> combination of TLOG+PULL replicas and utilising the solr autoscaling
> feature to maintain the stability of the cluster.
>
> I'm getting confused though with the documentation.
>
> When creating a collection, if you specify "replicationFactor=3" then it'll
> create 3 NRT replicas, and from my understanding, this information is
> preserved within autoscaling, i.e. if a node went down, it would attempt to
> add another replica to preserve the replicationFactor.
>
> However, because I'm using TLOG + PULL, I don't add the
> "replicationFactor=3" into my collection creation, otherwise it ends up
> creating a collection with 3 NRT replicas, 1 TLOG and 2 PULL replicas,
> which of course I do not want.
>
> Instead, I add "=1=2" which satisfies. However,
> when it comes to me creating policies, I'm not getting violations when
> expected.
>
> I created the following policies:
> {
>   "replica": "<2",
>   "shard": "#EACH",
>   "type": "TLOG",
>   "node": "#ANY"
> },
> {
>   "replica": "<3",
>   "shard": "#EACH",
>   "type": "PULL",
>   "node": "#ANY"
> }
>
> And there appears to be 0 violations, however, when I add another replica,
> there are still 0 violations despite there now being 4 replicas for a given
> shard of a collection;
>
> "violations":[],
> "config":{
>   "cluster-preferences":[{
>   "minimize":"cores",
>   "precision":1}
> ,{
>   "maximize":"freedisk"}],
>   "cluster-policy":[{
>   "replica":"<2",
>   "shard":"#EACH",
>   "type":"TLOG",
>   "node":"#ANY"}
> ,{
>   "replica":"<3",
>   "shard":"#EACH",
>   "type":"PULL",
>   "node":"#ANY"}]}},
>
> And here is the information about a collection-shard in which I added an
> extra replica to which I expected there to be a violation:
> {
>   "replica_p35": {
> "core": "collection_four_shard1_replica_p35",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_p31": {
> "core": "collection_four_shard1_replica_p31",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_p87": {
> "core": "collection_four_shard1_replica_p87",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_t75": {
> "core": "collection_four_shard1_replica_t75",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "type": "TLOG",
> "leader": "true"
>   }
> }
>
> Am I missing something to preserve replicationFactor for a collection, and
> trying to make collections have 1 TLOG replica and 2 PULL replicas?
>
> I tried adding the following
> {"replicas": "<4", "shard":"#EACH", "node": "#ANY"}
>
> However, still no luck
>
> Equally, how would I then go about setting up triggers to only create a
> PULL if a PULL goes down, and equally if the TLOG goes down? Would having a
> trigger for each type be needed?
>
> Any guidance on this would be greatly appreciated
> Thanks,
> --
>
> Richard Goodman
>

auotscaling tlog+pull and replicationFactor

2020-04-02 Thread Richard Goodman 
Hi there,

I'm currently using solr v7.7.2 *(I applied the patch SOLR-13674
 to my build to preserve
replica types for triggers)* and trying to set up a cluster with the
combination of TLOG+PULL replicas and utilising the solr autoscaling
feature to maintain the stability of the cluster.

I'm getting confused though with the documentation.

When creating a collection, if you specify "replicationFactor=3" then it'll
create 3 NRT replicas, and from my understanding, this information is
preserved within autoscaling, i.e. if a node went down, it would attempt to
add another replica to preserve the replicationFactor.

However, because I'm using TLOG + PULL, I don't add the
"replicationFactor=3" into my collection creation, otherwise it ends up
creating a collection with 3 NRT replicas, 1 TLOG and 2 PULL replicas,
which of course I do not want.

Instead, I add "=1=2" which satisfies. However,
when it comes to me creating policies, I'm not getting violations when
expected.

I created the following policies:
{
  "replica": "<2",
  "shard": "#EACH",
  "type": "TLOG",
  "node": "#ANY"
},
{
  "replica": "<3",
  "shard": "#EACH",
  "type": "PULL",
  "node": "#ANY"
}

And there appears to be 0 violations, however, when I add another replica,
there are still 0 violations despite there now being 4 replicas for a given
shard of a collection;

"violations":[],
"config":{
  "cluster-preferences":[{
  "minimize":"cores",
  "precision":1}
,{
  "maximize":"freedisk"}],
  "cluster-policy":[{
  "replica":"<2",
  "shard":"#EACH",
  "type":"TLOG",
  "node":"#ANY"}
,{
  "replica":"<3",
  "shard":"#EACH",
  "type":"PULL",
  "node":"#ANY"}]}},

And here is the information about a collection-shard in which I added an
extra replica to which I expected there to be a violation:
{
  "replica_p35": {
"core": "collection_four_shard1_replica_p35",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_p31": {
"core": "collection_four_shard1_replica_p31",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_p87": {
"core": "collection_four_shard1_replica_p87",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_t75": {
"core": "collection_four_shard1_replica_t75",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"type": "TLOG",
"leader": "true"
  }
}

Am I missing something to preserve replicationFactor for a collection, and
trying to make collections have 1 TLOG replica and 2 PULL replicas?

I tried adding the following
{"replicas": "<4", "shard":"#EACH", "node": "#ANY"}

However, still no luck

Equally, how would I then go about setting up triggers to only create a
PULL if a PULL goes down, and equally if the TLOG goes down? Would having a
trigger for each type be needed?

Any guidance on this would be greatly appreciated
Thanks,
-- 

Richard Goodman