[ https://issues.apache.org/jira/browse/AMBARI-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290572#comment-16290572 ]
Swapan Shridhar edited comment on AMBARI-22649 at 12/14/17 9:00 AM: -------------------------------------------------------------------- *Testing:* Tested on live cluster *=================* *clusterSettings:* *=================* *A.* get_cluster_setting_entries(): *------------------------------------* - 1. Retrieve *single* setting : 'recovery_enabled' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled']) *o/p*: {'recovery_enabled': True} - 2. Retrieve *two* settings : 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 3. Retrieve settings where passed in empty set -> Expected nothing returned -- In get_cluster_setting_entries(). Passed-in setting(s) : set([]) *o/p*: None - 4. Retrieve *three* settings : 'smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False, 'smokeuser': 'ambari-qa'} - 5. Retrieve *three* settings where *middle setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'abc', 'sysprep_skip_setup_jce']) *o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 6. Retrieve *three* settings where *1st setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['abc', 'recovery_enabled', 'sysprep_skip_setup_jce']) *o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 7. Retrieve *three* settings where *last setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce', 'abc']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 8. Retrieve passed in setting which is *non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1']) *o/p* : None - 9. Retrieve *two* passed in settings and both are non-existent -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1', 'non-existent2']) *o/p* : None - 10. Retrieve settings where set passed in is None. -> *returns all settings.* -- In get_cluster_setting_entries(). Passed-in setting(s) : None *o/p*: {'security_enabled': 'false', 'namenode_rolling_restart_timeout': '4200', 'enable_external_ranger': 'false', 'override_uid': 'true', 'kerberos_domain': 'EXAMPLE.COM', 'one_dir_per_partition': 'false', 'agent_mounts_ignore_list': '', 'repo_ubuntu_template': '{{package_type}} {{base_url}} {{components}}', 'ignore_groupsusers_create': 'false', 'alerts_repeat_tolerance': '1', 'hide_yarn_memory_widget': 'false', 'fetch_nonlocal_groups': 'true', 'manage_dirs_on_root': 'true', 'recovery_lifetime_max_count': '1024', 'recovery_type': 'AUTO_START', 'ignore_bad_mounts': 'false', 'recovery_window_in_minutes': '60', 'sysprep_skip_copy_tarballs_hdfs': 'false', 'user_group': 'hadoop', 'namenode_rolling_restart_safemode_exit_timeout': '3600', 'recovery_retry_interval': '5', 'sysprep_skip_copy_oozie_share_lib_to_hdfs': 'false', 'sysprep_skip_setup_jce': 'false', 'manage_hive_fsroot': 'true', 'service_check_type': 'full', 'recovery_enabled': 'true', 'recovery_max_count': '6', 'sysprep_skip_create_users_and_groups': 'false', 'smokeuser_keytab': '/etc/security/keytabs/smokeuser.headless.keytab', 'managed_hdfs_resource_property_names': 'false', 'smokeuser': 'ambari-qa', 'sysprep_skip_copy_fast_jar_hdfs': 'false'} *B.* get_cluster_setting_value(): *------------------------------------* - 1. Retrieve cluster_setting : 'hide_yarn_memory_widget' -- In get_cluster_setting_value(). Passed-in setting : hide_yarn_memory_widget *o/p* : false - 2. Retrieve cluster_setting : 'recovery_max_count' -- In get_cluster_setting_value(). Passed-in setting : recovery_max_count *o/p* : 6 - 3. Retrieve cluster_setting where passed in setting is 'non_existing' --> Empty string -- In get_cluster_setting_value(). Passed-in setting : non_existing *o/p* : None - 4. Retrieve cluster_setting setting passed in is "None" -- In get_cluster_setting_value(). Passed-in setting : None *o/p* : None *C*. is_security_enabled(): *------------------------------------* - 1. Retrieve security state of cluster by calling 'is_security_enabled()' which in turn calls *get_cluster_setting_value()* -- In get_cluster_setting_value(). Passed-in setting : security_enabled *o/p* : false *=================* *stackSettings*: *=================* *A.* Tests for *get_stack_setting_entries()* *------------------------------------* - 1. Retrieve : *stack_name* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name']) o/p: {'stack_name': 'HDP'} - 2. Retrieve : *stack_root* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_root']) *o/p*: {'stack_root': '{"HDP":"/usr/hdp"}'} - 3. Retrieve *three* stack settings: *'stack_name', 'stack_root', 'stack_tools'* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) *o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} - 4. Retrieve *four* stack settings : *'BAD_VALUE', 'stack_name', 'stack_root', 'stack_tools'* ---> (*BAD_VALUE* should get ignored) -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) *o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} *B.* Tests for existing and modified : *stack_select.get_packages()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "HDFS", "DATANODE") -- In get_stack_setting_value(). Passed-in setting : stack_packages *o/p* : ['hadoop-hdfs-datanode'] - 2. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "ZOOKEEPER_CLIENT") -- In get_stack_setting_value(). Passed-in setting : stack_packages *o/p* : ['zookeeper-client'] - 3. Retrieve *non-existing component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "NON_EXISTING") -- In get_stack_setting_value(). Passed-in setting : stack_packages Skipping stack-select on ABC because it does not exist in the stack-select package structure. *o/p* : None *C*. Tests for existing and modified : *stack_features.check_stack_feature()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.6") -- In get_stack_setting_value(). Passed-in setting : stack_features *o/p* : True - 2. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.2") -- In get_stack_setting_value(). Passed-in setting : stack_features *o/p* : False *D*. Test for existing and modified : *stack_tools.get_stack_tool()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *stack relevant tools* with call -> stack_tools.get_stack_tool("stack_selector") -- In get_stack_setting_value(). Passed-in setting : stack_tools *o/p*: ('hdp-select', '/usr/bin/hdp-select', 'hdp-select') was (Author: swapanshridhar): *Testing:* Tested on live cluster *=================* *clusterSettings:* *=================* *A.* get_cluster_setting_entries(): *------------------------------------* - 1. Retrieve *single* setting : 'recovery_enabled' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled']) *o/p*: {'recovery_enabled': True} - 2. Retrieve *two* settings : 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 3. Retrieve settings where passed in empty set -> Expected nothing returned -- In get_cluster_setting_entries(). Passed-in setting(s) : set([]) *o/p*: None - 4. Retrieve *three* settings : 'smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False, 'smokeuser': 'ambari-qa'} - 5. Retrieve *three* settings where *middle setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'abc', 'sysprep_skip_setup_jce']) *o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 6. Retrieve *three* settings where *1st setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['abc', 'recovery_enabled', 'sysprep_skip_setup_jce']) *o/p* : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 7. Retrieve *three* settings where *last setting is non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce', 'abc']) *o/p*: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 8. Retrieve passed in setting which is *non-existent* -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1']) *o/p* : None - 9. Retrieve *two* passed in settings and both are non-existent -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1', 'non-existent2']) *o/p* : None - 10. Retrieve settings where set passed in is None. -> *returns all settings.* -- In get_cluster_setting_entries(). Passed-in setting(s) : None *o/p*: {'security_enabled': 'false', 'namenode_rolling_restart_timeout': '4200', 'enable_external_ranger': 'false', 'override_uid': 'true', 'kerberos_domain': 'EXAMPLE.COM', 'one_dir_per_partition': 'false', 'agent_mounts_ignore_list': '', 'repo_ubuntu_template': '{{package_type}} {{base_url}} {{components}}', 'ignore_groupsusers_create': 'false', 'alerts_repeat_tolerance': '1', 'hide_yarn_memory_widget': 'false', 'fetch_nonlocal_groups': 'true', 'manage_dirs_on_root': 'true', 'recovery_lifetime_max_count': '1024', 'recovery_type': 'AUTO_START', 'ignore_bad_mounts': 'false', 'recovery_window_in_minutes': '60', 'sysprep_skip_copy_tarballs_hdfs': 'false', 'user_group': 'hadoop', 'namenode_rolling_restart_safemode_exit_timeout': '3600', 'recovery_retry_interval': '5', 'sysprep_skip_copy_oozie_share_lib_to_hdfs': 'false', 'sysprep_skip_setup_jce': 'false', 'manage_hive_fsroot': 'true', 'service_check_type': 'full', 'recovery_enabled': 'true', 'recovery_max_count': '6', 'sysprep_skip_create_users_and_groups': 'false', 'smokeuser_keytab': '/etc/security/keytabs/smokeuser.headless.keytab', 'managed_hdfs_resource_property_names': 'false', 'smokeuser': 'ambari-qa', 'sysprep_skip_copy_fast_jar_hdfs': 'false'} *B.* get_cluster_setting_value(): *------------------------------------* - 1. Retrieve cluster_setting : 'hide_yarn_memory_widget' -- In get_cluster_setting_value(). Passed-in setting : hide_yarn_memory_widget *o/p* : false - 2. Retrieve cluster_setting : 'recovery_max_count' -- In get_cluster_setting_value(). Passed-in setting : recovery_max_count *o/p* : 6 - 3. Retrieve cluster_setting where passed in setting is 'non_existing' --> Empty string -- In get_cluster_setting_value(). Passed-in setting : non_existing *o/p* : None - 4. Retrieve cluster_setting setting passed in is "None" -- In get_cluster_setting_value(). Passed-in setting : None *o/p* : None *C*. is_security_enabled(): *------------------------------------* - 1. Retrieve security state of cluster by calling 'is_security_enabled()' which in turn calls *get_cluster_setting_value()* -- In get_cluster_setting_value(). Passed-in setting : security_enabled *o/p* : false *=================* *stackSettings*: *=================* *A.* Tests for *get_stack_setting_entries()* *------------------------------------* - 1. Retrieve : *stack_name* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name']) o/p: {'stack_name': 'HDP'} - 2. Retrieve : *stack_root* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_root']) *o/p*: {'stack_root': '{"HDP":"/usr/hdp"}'} - 3. Retrieve *three* stack settings: *'stack_name', 'stack_root', 'stack_tools'* -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) *o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} - 4. Retrieve *four* stack settings : *'BAD_VALUE', 'stack_name', 'stack_root', 'stack_tools'* ---> (*BAD_VALUE* should get ignored) -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) *o/p* : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} *B.* Tests for existing and modified : *stack_select.get_packages()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "HDFS", "DATANODE") -- In get_stack_setting_value(). Passed-in setting : stack_packages *o/p* : ['hadoop-hdfs-datanode'] - 2. Retrieve *specific component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "ZOOKEEPER_CLIENT") -- In get_stack_setting_value(). Passed-in setting : stack_packages *o/p* : ['zookeeper-client'] - 3. Retrieve *non-existing component package* with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "NON_EXISTING") -- In get_stack_setting_value(). Passed-in setting : stack_packages Skipping stack-select on ABC because it does not exist in the stack-select package structure. *o/p* : None *C*. Tests for existing and modified : *stack_features.check_stack_feature()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.6") -- In get_stack_setting_value(). Passed-in setting : stack_features *o/p* : True - 2. Retrieve *specific component stack feature* with call -> stack_features.check_stack_feature("hive_server_interactive", "2.2") -- In get_stack_setting_value(). Passed-in setting : stack_features *o/p* : False *D*. Test for existing and modified : *stack_tools.get_stack_tool()* which internally now calls *stack_settings.get_stack_setting_value()* *------------------------------------------------------------------------------------------------------------------------------------------------------------* - 1. Retrieve *stack relevant tools* with call -> stack_tools.get_stack_tool("stack_selector") -- In get_stack_setting_value(). Passed-in setting : stack_tools *o/p*: ('hdp-select', '/usr/bin/hdp-select', 'hdp-select') CC [~jluniya] | [~bsari] | [~mradhakrishnan] > Library for querying cluster_settings and stack_settings in command*.json. > -------------------------------------------------------------------------- > > Key: AMBARI-22649 > URL: https://issues.apache.org/jira/browse/AMBARI-22649 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Reporter: Swapan Shridhar > Assignee: Swapan Shridhar > Fix For: 3.0.0 > > Attachments: AMBARI-22649.patch > > > Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced > "cluster settings" in Ambari. > *=========================================================================* > *Library for querying _clusterSettings_ and _stackSettings_ for its contents > in command\*.json.* > *=========================================================================* > One should be able to query for a given *clusterSettings* or *stackSettings*: > - by passing in the setting name(one or more) in order to get it back as > key-value map, or > - just get the value back for a passed-in setting. > *Functions for clusterSettings:* > *-----------------------------------------* > - get_cluster_setting_entries(setting_names) : > -- Retrieves the passed-in cluster setting entr(y/ies) and their values > as a map. > If 'setting_names' is passed-in as None : all the settings names and > their corresponding values will be returned as map. > If 'setting_names' is passed-in as empty set : None will be returned. > - get_cluster_setting_value(setting_name) : > -- Retrieves the passed-in cluster setting entry's value. > - is_security_enabled() : > -- Retrieves the cluster's security status. > *Functions for stackSettings:* > *-----------------------------------------* > Stack settings as of now has 5 settings : stack_name, stack_root, > stack_features, stack_tools, stack_packages. stack_name, stack_root have > string as values, whereas stack_features, stack_tools, stack_packages have > values as JSON. Further there already exists python functions in files : > *stack_features.py*, *stack_tools.py* and *stack_select.py*. > - get_stack_setting_entries(setting_names) : > -- Retrieves the passed-in stack setting entr(y/ies) and their values > as a map. > If 'setting_names' is passed-in as None, all the settings names > and their corresponding values will be returned as map. > If 'setting_names' is passed-in as empty set : None will be > returned. > - get_stack_setting_value(setting_name): > -- Retrieves the passed-in stack setting entry's value. > - get_stack_name(): > -- Retrieves the stack name. > - get_stack_root(): > -- Retrieves the stack root. > > *Modifications in _stack_features.py, stack_tools.py and stack_select.py_ > files:* > *--------------------------------------------------------------------------------------------------------------* > - Given that these already exist and as of now they read the relevant stack > setting from *configurations/cluster_env*. > - Thus, code has been added to try reading from /stackSettings first by > calling the new fn.() get_stack_setting_value(). if setting not found, go for > the fall back *configurations/cluster_env* (which would be removed soon, > when we remove cluster_env). -- This message was sent by Atlassian JIRA (v6.4.14#64029)