Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
Entered ticket https://issues.apache.org/jira/browse/SOLR-13586
Sadly, no patch attached this time as it is a much more complicated issue than
my last one, and a good bit above my paygrade with Java.
From: Andrzej Białecki
Sent: Friday, June 28, 2019 4:29:49 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules,
possibly all rules
Andrew, please create a JIRA issue - in my opinion this is a bug not a feature,
or at least something that needs clarification.
> On 27 Jun 2019, at 23:56, Andrew Kettmann
> wrote:
>
> I found the issue. Autoscaling seems to silently ignore rules (at least
> sysprop rules). Example rule:
>
>
> {'set-policy': {'sales-uat': [{'node': '#ANY',
> 'replica': '<2',
> 'strict': 'false'},
> {'replica': '#ALL',
> 'strict': 'true',
> 'sysprop.HELM_CHART': 'foo'}]}}
>
>
> Two cases will get the sysprop rule ignored:
>
> 1. No nodes have a HELM_CHART system property defined
> 2. No nodes have the value "foo" for the HELM_CHART system property
>
>
> If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it
> cannot satisfy another strict rule. So sysprop autoscaling rules appear to be
> unable to be strict on their own it appears.
>
>
> Hopefully this can solve some issues for other people as well.
>
>
> From: Andrew Kettmann
> Sent: Tuesday, June 25, 2019 1:04:21 PM
> To: solr-user@lucene.apache.org
> Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules,
> possibly all rules
>
>
> Using docker 7.7.2 image
>
>
> Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
>
>
> Created a policy:
>
> {'set-policy': {'banana': [{'replica': '#ALL',
>'sysprop.HELM_CHART': 'notbanana'}]}}
>
>
> No errors on creation of the policy.
>
>
> I have no nodes that have that value for the system property "HELM_CHART", I
> have nodes that contain "banana" and "rulesos" for that value only.
>
>
> I create the collection with a call to the /admin/collections:
>
> {'action': 'CREATE',
> 'collection.configName': 'project-solr-7',
> 'name': 'banana',
> 'numShards': '2',
> 'policy': 'banana',
> 'replicationFactor': '2'}
>
>
> and it creates the collection without an error. Which what I expected was the
> collection creation to fail. This is the behavior I had seen in the past, but
> after tearing down and recreating the cluster in a higher environment, it
> does not appear to function.
>
>
> Is there some prerequisite before policies will be respected? The .system
> collection is in place as expected, and I am not seeing anything in the logs
> on the overseer to suggest any problems.
>
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
> Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter]
> <https://twitter.com/evolve24> [Instagram]
> <https://www.instagram.com/evolve_24>
>
> evolve24 Confidential & Proprietary Statement: This email and any attachments
> are confidential and may contain information that is privileged, confidential
> or exempt from disclosure under applicable law. It is intended for the use of
> the recipients. If you are not the intended recipient, or believe that you
> have received this communication in error, please do not read, print, copy,
> retransmit, disseminate, or otherwise use the information. Please delete this
> email and attachments, without reading, printing, copying, forwarding or
> saving them, and notify the Sender immediately by reply email. No
> confidentiality or privilege is waived or lost by any transmission in error.
Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
Andrew, please create a JIRA issue - in my opinion this is a bug not a feature,
or at least something that needs clarification.
> On 27 Jun 2019, at 23:56, Andrew Kettmann
> wrote:
>
> I found the issue. Autoscaling seems to silently ignore rules (at least
> sysprop rules). Example rule:
>
>
> {'set-policy': {'sales-uat': [{'node': '#ANY',
> 'replica': '<2',
> 'strict': 'false'},
> {'replica': '#ALL',
> 'strict': 'true',
> 'sysprop.HELM_CHART': 'foo'}]}}
>
>
> Two cases will get the sysprop rule ignored:
>
> 1. No nodes have a HELM_CHART system property defined
> 2. No nodes have the value "foo" for the HELM_CHART system property
>
>
> If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it
> cannot satisfy another strict rule. So sysprop autoscaling rules appear to be
> unable to be strict on their own it appears.
>
>
> Hopefully this can solve some issues for other people as well.
>
>
> From: Andrew Kettmann
> Sent: Tuesday, June 25, 2019 1:04:21 PM
> To: solr-user@lucene.apache.org
> Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules,
> possibly all rules
>
>
> Using docker 7.7.2 image
>
>
> Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
>
>
> Created a policy:
>
> {'set-policy': {'banana': [{'replica': '#ALL',
>'sysprop.HELM_CHART': 'notbanana'}]}}
>
>
> No errors on creation of the policy.
>
>
> I have no nodes that have that value for the system property "HELM_CHART", I
> have nodes that contain "banana" and "rulesos" for that value only.
>
>
> I create the collection with a call to the /admin/collections:
>
> {'action': 'CREATE',
> 'collection.configName': 'project-solr-7',
> 'name': 'banana',
> 'numShards': '2',
> 'policy': 'banana',
> 'replicationFactor': '2'}
>
>
> and it creates the collection without an error. Which what I expected was the
> collection creation to fail. This is the behavior I had seen in the past, but
> after tearing down and recreating the cluster in a higher environment, it
> does not appear to function.
>
>
> Is there some prerequisite before policies will be respected? The .system
> collection is in place as expected, and I am not seeing anything in the logs
> on the overseer to suggest any problems.
>
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
> Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter]
> <https://twitter.com/evolve24> [Instagram]
> <https://www.instagram.com/evolve_24>
>
> evolve24 Confidential & Proprietary Statement: This email and any attachments
> are confidential and may contain information that is privileged, confidential
> or exempt from disclosure under applicable law. It is intended for the use of
> the recipients. If you are not the intended recipient, or believe that you
> have received this communication in error, please do not read, print, copy,
> retransmit, disseminate, or otherwise use the information. Please delete this
> email and attachments, without reading, printing, copying, forwarding or
> saving them, and notify the Sender immediately by reply email. No
> confidentiality or privilege is waived or lost by any transmission in error.
Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
I found the issue. Autoscaling seems to silently ignore rules (at least sysprop
rules). Example rule:
{'set-policy': {'sales-uat': [{'node': '#ANY',
'replica': '<2',
'strict': 'false'},
{'replica': '#ALL',
'strict': 'true',
'sysprop.HELM_CHART': 'foo'}]}}
Two cases will get the sysprop rule ignored:
1. No nodes have a HELM_CHART system property defined
2. No nodes have the value "foo" for the HELM_CHART system property
If you have SOME nodes that have -DHELM_CHART=foo, then it will fail if it
cannot satisfy another strict rule. So sysprop autoscaling rules appear to be
unable to be strict on their own it appears.
Hopefully this can solve some issues for other people as well.
________________
From: Andrew Kettmann
Sent: Tuesday, June 25, 2019 1:04:21 PM
To: solr-user@lucene.apache.org
Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules,
possibly all rules
Using docker 7.7.2 image
Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
Created a policy:
{'set-policy': {'banana': [{'replica': '#ALL',
'sysprop.HELM_CHART': 'notbanana'}]}}
No errors on creation of the policy.
I have no nodes that have that value for the system property "HELM_CHART", I
have nodes that contain "banana" and "rulesos" for that value only.
I create the collection with a call to the /admin/collections:
{'action': 'CREATE',
'collection.configName': 'project-solr-7',
'name': 'banana',
'numShards': '2',
'policy': 'banana',
'replicationFactor': '2'}
and it creates the collection without an error. Which what I expected was the
collection creation to fail. This is the behavior I had seen in the past, but
after tearing down and recreating the cluster in a higher environment, it does
not appear to function.
Is there some prerequisite before policies will be respected? The .system
collection is in place as expected, and I am not seeing anything in the logs on
the overseer to suggest any problems.
[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter]
<https://twitter.com/evolve24> [Instagram]
<https://www.instagram.com/evolve_24>
evolve24 Confidential & Proprietary Statement: This email and any attachments
are confidential and may contain information that is privileged, confidential
or exempt from disclosure under applicable law. It is intended for the use of
the recipients. If you are not the intended recipient, or believe that you have
received this communication in error, please do not read, print, copy,
retransmit, disseminate, or otherwise use the information. Please delete this
email and attachments, without reading, printing, copying, forwarding or saving
them, and notify the Sender immediately by reply email. No confidentiality or
privilege is waived or lost by any transmission in error.
Re: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
Is there some step I am missing here? Policies seem to be entirely ignored in
this new cluster and I am at a loss. Is there some default setting that will
cause autoscaling to be ignored?
From: Andrew Kettmann
Sent: Tuesday, June 25, 2019 1:04:21 PM
To: solr-user@lucene.apache.org
Subject: Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules,
possibly all rules
Using docker 7.7.2 image
Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
Created a policy:
{'set-policy': {'banana': [{'replica': '#ALL',
'sysprop.HELM_CHART': 'notbanana'}]}}
No errors on creation of the policy.
I have no nodes that have that value for the system property "HELM_CHART", I
have nodes that contain "banana" and "rulesos" for that value only.
I create the collection with a call to the /admin/collections:
{'action': 'CREATE',
'collection.configName': 'project-solr-7',
'name': 'banana',
'numShards': '2',
'policy': 'banana',
'replicationFactor': '2'}
and it creates the collection without an error. Which what I expected was the
collection creation to fail. This is the behavior I had seen in the past, but
after tearing down and recreating the cluster in a higher environment, it does
not appear to function.
Is there some prerequisite before policies will be respected? The .system
collection is in place as expected, and I am not seeing anything in the logs on
the overseer to suggest any problems.
[https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn]<https://linkedin.com/company/evolve24> [Twitter]
<https://twitter.com/evolve24> [Instagram]
<https://www.instagram.com/evolve_24>
evolve24 Confidential & Proprietary Statement: This email and any attachments
are confidential and may contain information that is privileged, confidential
or exempt from disclosure under applicable law. It is intended for the use of
the recipients. If you are not the intended recipient, or believe that you have
received this communication in error, please do not read, print, copy,
retransmit, disseminate, or otherwise use the information. Please delete this
email and attachments, without reading, printing, copying, forwarding or saving
them, and notify the Sender immediately by reply email. No confidentiality or
privilege is waived or lost by any transmission in error.
Solr 7.7.2 - Autoscaling in new cluster ignoring sysprop rules, possibly all rules
Using docker 7.7.2 image
Solr 7.7.2 on new Znode on ZK. Created the chroot using solr zk mkroot.
Created a policy:
{'set-policy': {'banana': [{'replica': '#ALL',
'sysprop.HELM_CHART': 'notbanana'}]}}
No errors on creation of the policy.
I have no nodes that have that value for the system property "HELM_CHART", I
have nodes that contain "banana" and "rulesos" for that value only.
I create the collection with a call to the /admin/collections:
{'action': 'CREATE',
'collection.configName': 'project-solr-7',
'name': 'banana',
'numShards': '2',
'policy': 'banana',
'replicationFactor': '2'}
and it creates the collection without an error. Which what I expected was the
collection creation to fail. This is the behavior I had seen in the past, but
after tearing down and recreating the cluster in a higher environment, it does
not appear to function.
Is there some prerequisite before policies will be respected? The .system
collection is in place as expected, and I am not seeing anything in the logs on
the overseer to suggest any problems.
[https://storage.googleapis.com/e24-email-images/e24logonotag.png]
Andrew Kettmann
DevOps Engineer
P: 1.314.596.2836
[LinkedIn] [Twitter]
[Instagram]
evolve24 Confidential & Proprietary Statement: This email and any attachments
are confidential and may contain information that is privileged, confidential
or exempt from disclosure under applicable law. It is intended for the use of
the recipients. If you are not the intended recipient, or believe that you have
received this communication in error, please do not read, print, copy,
retransmit, disseminate, or otherwise use the information. Please delete this
email and attachments, without reading, printing, copying, forwarding or saving
them, and notify the Sender immediately by reply email. No confidentiality or
privilege is waived or lost by any transmission in error.
