[
https://issues.apache.org/jira/browse/METRON-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated METRON-865:
------------------------------
Description:
Multiple bug fixes and recommended improvements were found in the course of
implementing METRON-608 that are unrelated to METRON-609 (singlenode install).
Almost all items relate to Elasticsearch.
About half the fixes did not impact the Ambari database, and were done in
METRON-634 (PR#532).
This jira provides the work items for changes that do impact the Ambari
database, and should therefore be done with an Mpack version bump and database
upgrade script. Implementation of these may be seen in the closed PR
https://github.com/apache/metron/pull/425
h2. Affects Ambari database, needs db upgrade script:
* status_params.py, which redundantly defines pid_dir as a python variable, is
unnecessary and unused by the ES portion of the Mpack. It can be removed.
* pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
elastic-env.xml, as it is a parameter that must be provided to ES at
launch-time, but is not something there's any reason for the admin to change in
usual circumstances.
* conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
elastic-sysconfig.xml. While it too is a parameter that must be provided to ES
at launch-time, it is typically left to the installing admin where to put the
config files.
* The configuration parameter names in elastic-site.xml should be improved in
several instances to make the semantics more obvious to the human reader (who
may not be real familiar with Elasticsearch configuration). Mouse-over
documentation will continue to provide the ES config parameter equivalents. In
particular, suggest:
{code}
cluster_name -> es_cluster_name (to distinguish ES cluster from Stack cluster)
zen_discovery_ping_unicast_hosts -> es_cluster_hosts
network_host -> network_bindings (these are in fact interface names, not host
names)
{code}
* This item moved to METRON-905 : -There are at least two places in
elasticsearch.master.yaml.j2 (zen_discovery_ping_unicast_hosts and
network_host) where needed square brackets are either missing or included in
the configuration string. To be consistent with other usages, and less prone
to human error, the square brackets should not be in the configuration string
but rather should be provided in the template text.-
* "data_dir" apparently should be eliminated (from elastic-sysconfig) in
preference for "path_data" (in elastic-site.xml). The latter value ends up
overriding the former anyway, but the existence of the former is confusing and
unnecessary.
* All four configuration parameters in elastic-env.xml should be moved to
elastic-site.xml, because they are all reasonable to set in a "site.xml" file
and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
end up in environment variables. The only parameters that end up in env vars
are set in elastic-sysconfig, and the ES launch process in fact ignores the
elastic-env.sh file that is templated in elastic-env.xml (which consists only
of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
hence entirely remove elastic-env.xml, or we could choose to keep the small
elastic-env.sh file and its template, just to remind people that it is
necessary to have JAVA_HOME defined.
* In METRON/0.3.0/configuration/metron-env.xml and
METRON/0.3.0/package/scripts/params/params_linux.py, the value
"metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus it
can be removed from metron-env.xml and set to
"\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
eliminates a really unacceptable use of "double format".
was:
Multiple bug fixes and recommended improvements were found in the course of
implementing METRON-608 that are unrelated to METRON-609 (singlenode install).
Almost all items relate to Elasticsearch.
About half the fixes did not impact the Ambari database, and were done in
METRON-634 (PR#532).
This jira provides the work items for changes that do impact the Ambari
database, and should therefore be done with an Mpack version bump and database
upgrade script.
h2. Affects Ambari database, needs db upgrade script:
* status_params.py, which redundantly defines pid_dir as a python variable, is
unnecessary and unused by the ES portion of the Mpack. It can be removed.
* pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
elastic-env.xml, as it is a parameter that must be provided to ES at
launch-time, but is not something there's any reason for the admin to change in
usual circumstances.
* conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
elastic-sysconfig.xml. While it too is a parameter that must be provided to ES
at launch-time, it is typically left to the installing admin where to put the
config files.
* The configuration parameter names in elastic-site.xml should be improved in
several instances to make the semantics more obvious to the human reader (who
may not be real familiar with Elasticsearch configuration). Mouse-over
documentation will continue to provide the ES config parameter equivalents. In
particular, suggest:
{code}
cluster_name -> es_cluster_name (to distinguish ES cluster from Stack cluster)
zen_discovery_ping_unicast_hosts -> es_cluster_hosts
network_host -> network_bindings (these are in fact interface names, not host
names)
{code}
* This item moved to METRON-905 : -There are at least two places in
elasticsearch.master.yaml.j2 (zen_discovery_ping_unicast_hosts and
network_host) where needed square brackets are either missing or included in
the configuration string. To be consistent with other usages, and less prone
to human error, the square brackets should not be in the configuration string
but rather should be provided in the template text.-
* "data_dir" apparently should be eliminated (from elastic-sysconfig) in
preference for "path_data" (in elastic-site.xml). The latter value ends up
overriding the former anyway, but the existence of the former is confusing and
unnecessary.
* All four configuration parameters in elastic-env.xml should be moved to
elastic-site.xml, because they are all reasonable to set in a "site.xml" file
and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
end up in environment variables. The only parameters that end up in env vars
are set in elastic-sysconfig, and the ES launch process in fact ignores the
elastic-env.sh file that is templated in elastic-env.xml (which consists only
of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
hence entirely remove elastic-env.xml, or we could choose to keep the small
elastic-env.sh file and its template, just to remind people that it is
necessary to have JAVA_HOME defined.
* In METRON/0.3.0/configuration/metron-env.xml and
METRON/0.3.0/package/scripts/params/params_linux.py, the value
"metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus it
can be removed from metron-env.xml and set to
"\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
eliminates a really unacceptable use of "double format".
> Additional Mpack bug fixes and improvements, that affect Ambari database
> ------------------------------------------------------------------------
>
> Key: METRON-865
> URL: https://issues.apache.org/jira/browse/METRON-865
> Project: Metron
> Issue Type: Sub-task
> Affects Versions: 0.3.0
> Environment: Centos7
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.3.1
>
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). Almost all items relate to Elasticsearch.
> About half the fixes did not impact the Ambari database, and were done in
> METRON-634 (PR#532).
> This jira provides the work items for changes that do impact the Ambari
> database, and should therefore be done with an Mpack version bump and
> database upgrade script. Implementation of these may be seen in the closed PR
> https://github.com/apache/metron/pull/425
> h2. Affects Ambari database, needs db upgrade script:
> * status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
> zen_discovery_ping_unicast_hosts -> es_cluster_hosts
> network_host -> network_bindings (these are in fact interface names, not
> host names)
> {code}
> * This item moved to METRON-905 : -There are at least two places in
> elasticsearch.master.yaml.j2 (zen_discovery_ping_unicast_hosts and
> network_host) where needed square brackets are either missing or included in
> the configuration string. To be consistent with other usages, and less prone
> to human error, the square brackets should not be in the configuration string
> but rather should be provided in the template text.-
> * "data_dir" apparently should be eliminated (from elastic-sysconfig) in
> preference for "path_data" (in elastic-site.xml). The latter value ends up
> overriding the former anyway, but the existence of the former is confusing
> and unnecessary.
> * All four configuration parameters in elastic-env.xml should be moved to
> elastic-site.xml, because they are all reasonable to set in a "site.xml" file
> and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
> end up in environment variables. The only parameters that end up in env vars
> are set in elastic-sysconfig, and the ES launch process in fact ignores the
> elastic-env.sh file that is templated in elastic-env.xml (which consists only
> of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
> hence entirely remove elastic-env.xml, or we could choose to keep the small
> elastic-env.sh file and its template, just to remind people that it is
> necessary to have JAVA_HOME defined.
> * In METRON/0.3.0/configuration/metron-env.xml and
> METRON/0.3.0/package/scripts/params/params_linux.py, the value
> "metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
> appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus
> it can be removed from metron-env.xml and set to
> "\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
> eliminates a really unacceptable use of "double format".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
