Cache fails to warm after Replication Recovery in solr cloud
Hi! I have some custom cache set up in solrconfig XML for a solr cloud cluster in Kubernetes. Each node has Kubernetes persistence set up. After I execute a “delete pod” command to restart a node it goes into Replication Recovery successfully but my custom cache’s warm() method never gets called. Is this expected behavior? The events I observed are: 1. Cache init() method called 2. Searcher created and registered 3. Replication recovery Thanks! Li
Re: [EXTERNAL] Autoscaling simulation error
Thank you for creating the JIRA! Will follow
On 12/19/19, 11:09 AM, "Andrzej Białecki" wrote:
Hi,
Thanks for the data. I see the problem now - it’s a bug in the simulator. I
filed a Jira issue to track and fix it: SOLR-14122.
> On 16 Dec 2019, at 19:13, Cao, Li wrote:
>
>> I am using solr 8.3.0 in cloud mode. I have collection level autoscaling
policy and the collection name is “entity”. But when I run autoscaling
simulation all the steps failed with this message:
>>
>> "error":{
>> "exception":"java.io.IOException:
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException:
org.apache.solr.common.SolrException: Could not find collection :
entity/shards",
>> "suggestion":{
>> "type":"repair",
>> "operation":{
>> "method":"POST",
>> "path":"/c/entity/shards",
>> "command":{"add-replica":{
>> "shard":"shard2",
>> "node":"my_node:8983_solr",
>> "type":"TLOG",
>> "replicaInfo":null}}},
Re: [EXTERNAL] Re: "No value present" when set cluster policy for autoscaling in solr cloud mode
Thank you, Andrzej! I am going to try IN operand as a work around.
On 12/19/19, 10:17 AM, "Andrzej Białecki" wrote:
Hi,
For some strange reason global tags (such as “cores”) don’t support the
“nodeset” syntax. For “cores” the only supported attribute is “node”, and then
you’re only allowed to use #ANY or a single specific node name (with optional
“!" NOT operand), or a JSON array containing node names to indicate the IN
operand.
The Ref Guide indeed is not very clear on that…
> On 17 Dec 2019, at 21:20, Cao, Li wrote:
>
> Hi!
>
> I am trying to add a cluster policy to a freshly built 8.3.0 cluster (no
collection added). I got this error when adding such a cluster policy
>
> {
"set-cluster-policy":[{"cores":"<3","nodeset":{"sysprop.rex.node.type":"tlog"}}]}
>
> Basically I want to limit the number of cores for certain machines with a
special environmental variable value.
>
> But I got this error response:
>
> {
> "responseHeader":{
>"status":400,
>"QTime":144},
> "result":"failure",
> "WARNING":"This response format is experimental. It is likely to change
in the future.",
> "error":{
>"metadata":[
> "error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject",
>
"root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],
>"details":[{
>"set-cluster-policy":[{
>"cores":"<3",
>"nodeset":{"sysprop.rex.node.type":"tlog"}}],
>"errorMessages":["No value present"]}],
>"msg":"Error in command payload",
>"code":400}}
>
> However, this works:
>
> { "set-cluster-policy":[{"cores":"<3","node":"#ANY"}]}
>
> I read the autoscaling policy documentations and cannot figure out why.
Could someone help me on this?
>
> Thanks!
>
> Li
"No value present" when set cluster policy for autoscaling in solr cloud mode
Hi!
I am trying to add a cluster policy to a freshly built 8.3.0 cluster (no
collection added). I got this error when adding such a cluster policy
{
"set-cluster-policy":[{"cores":"<3","nodeset":{"sysprop.rex.node.type":"tlog"}}]}
Basically I want to limit the number of cores for certain machines with a
special environmental variable value.
But I got this error response:
{
"responseHeader":{
"status":400,
"QTime":144},
"result":"failure",
"WARNING":"This response format is experimental. It is likely to change in
the future.",
"error":{
"metadata":[
"error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject",
"root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],
"details":[{
"set-cluster-policy":[{
"cores":"<3",
"nodeset":{"sysprop.rex.node.type":"tlog"}}],
"errorMessages":["No value present"]}],
"msg":"Error in command payload",
"code":400}}
However, this works:
{ "set-cluster-policy":[{"cores":"<3","node":"#ANY"}]}
I read the autoscaling policy documentations and cannot figure out why. Could
someone help me on this?
Thanks!
Li
Re: [EXTERNAL] Re: Autoscaling simulation error
Hi Andrzej ,
I have put the JSONs produced by "save" commands below:
autoscalingState.json - https://pastebin.com/CrR0TdLf
clusterState.json - https://pastebin.com/zxuYAMux
nodeState.json https://pastebin.com/hxqjVUfV
statistics.json https://pastebin.com/Jkaw8Y3j
The simulate command is:
/opt/solr-8.3.0/bin/solr autoscaling -a policy2.json -simulate -zkHost
rexcloud-swoods-zookeeper-headless:2181
Policy2 can be found here:
https://pastebin.com/VriJ27DE
Setup:
12 nodes on Kubernetes. 6 for TLOG and 6 for Pull. The simulation is run on one
of nodes inside Kubernetes because it needs the zookeeper inside the Kubernetes.
Thanks!
Li
On 12/15/19, 5:13 PM, "Andrzej Białecki" wrote:
Could you please provide the exact command-line? It would also help if you
could provide an autoscaling snapshot of the cluster (bin/solr autoscaling
-save ) or at least the autoscaling diagnostic info.
(Please note that the mailing list removes all attachments, so just provide
a link to the snapshot).
> On 15 Dec 2019, at 18:42, Cao, Li wrote:
>
> Hi!
>
> I am using solr 8.3.0 in cloud mode. I have collection level autoscaling
policy and the collection name is “entity”. But when I run autoscaling
simulation all the steps failed with this message:
>
>"error":{
> "exception":"java.io.IOException:
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException:
org.apache.solr.common.SolrException: Could not find collection :
entity/shards",
> "suggestion":{
>"type":"repair",
>"operation":{
> "method":"POST",
> "path":"/c/entity/shards",
> "command":{"add-replica":{
> "shard":"shard2",
> "node":"my_node:8983_solr",
> "type":"TLOG",
> "replicaInfo":null}}},
>
> Does anyone know how to fix this? Is this a bug?
>
> Thanks!
>
> Li
Autoscaling simulation error
Hi!
I am using solr 8.3.0 in cloud mode. I have collection level autoscaling policy
and the collection name is “entity”. But when I run autoscaling simulation all
the steps failed with this message:
"error":{
"exception":"java.io.IOException:
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException:
org.apache.solr.common.SolrException: Could not find collection :
entity/shards",
"suggestion":{
"type":"repair",
"operation":{
"method":"POST",
"path":"/c/entity/shards",
"command":{"add-replica":{
"shard":"shard2",
"node":"my_node:8983_solr",
"type":"TLOG",
"replicaInfo":null}}},
Does anyone know how to fix this? Is this a bug?
Thanks!
Li
