Repository: ambari Updated Branches: refs/heads/trunk c405aacf5 -> 76c1fc247
AMBARI-20220 - Parameterize the Startup Web Server Timeout Default Value (jonathanhurley) Project: http://git-wip-us.apache.org/repos/asf/ambari/repo Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/76c1fc24 Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/76c1fc24 Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/76c1fc24 Branch: refs/heads/trunk Commit: 76c1fc2473b311ea25512cbde7019b28f0b47ad9 Parents: c405aac Author: Jonathan Hurley <jhur...@hortonworks.com> Authored: Mon Feb 27 20:16:05 2017 -0500 Committer: Jonathan Hurley <jhur...@hortonworks.com> Committed: Tue Feb 28 08:56:34 2017 -0500 ---------------------------------------------------------------------- ambari-server/docs/configuration/index.md | 7 +++++- .../server/configuration/Configuration.java | 18 ++++++++++---- .../python/ambari_server/serverConfiguration.py | 22 +++++++++++++++++ .../src/main/python/ambari_server_main.py | 24 +++++++++--------- .../src/test/python/TestAmbariServer.py | 26 ++++++++++++++++++-- 5 files changed, 77 insertions(+), 20 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ambari/blob/76c1fc24/ambari-server/docs/configuration/index.md ---------------------------------------------------------------------- diff --git a/ambari-server/docs/configuration/index.md b/ambari-server/docs/configuration/index.md index ae2d549..968db56 100644 --- a/ambari-server/docs/configuration/index.md +++ b/ambari-server/docs/configuration/index.md @@ -44,6 +44,7 @@ The following are the properties which can be used to configure Ambari. | agent.check.remote.mounts | Determines whether the Ambari Agents will use the `df` or `df -l` command when checking disk mounts for capacity issues. Auto-mounted remote directories can cause long delays. |`false` | | agent.package.install.task.timeout | The time, in seconds, before package installation commands are killed. |`1800` | | agent.package.parallel.commands.limit | The maximum number of tasks which can run within a single operational request. If there are more tasks, then they will be broken up between multiple operations. |`100` | +| agent.service.check.task.timeout | The time, in seconds, before agent service check commands are killed. |`0` | | agent.ssl | Determines whether SSL is used to communicate between Ambari Server and Ambari Agents. |`true` | | agent.stack.retry.on_repo_unavailability | Determines whether agents should retrying installation commands when the repository is not available. This can prevent false installation errors with repositories that are sporadically inaccessible. |`false` | | agent.stack.retry.tries | The number of times an Ambari Agent should retry package installation when it fails due to a repository error. <br/><br/> This property is related to `agent.stack.retry.on_repo_unavailability`. |`5` | @@ -61,6 +62,7 @@ The following are the properties which can be used to configure Ambari. | ambari.post.user.creation.hook | The location of the post user creation hook on the ambari server hosting machine. |`/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh` | | ambari.post.user.creation.hook.enabled | Indicates whether the post user creation is enabled or not. By default is false. |`false` | | ambari.python.wrap | The name of the shell script used to wrap all invocations of Python by Ambari. |`ambari-python-wrap` | +| ambariserver.metrics.disable | Global disable flag for AmbariServer Metrics. |`false` | | anonymous.audit.name | The name of the user given to requests which are executed without any credentials. |`_anonymous` | | api.authenticated.user | The username of the default user assumed to be executing API calls. When set, authentication is not required in order to login to Ambari or use the REST APIs. | | | api.csrfPrevention.enabled | Determines whether Cross-Site Request Forgery attacks are prevented by looking for the `X-Requested-By` header. |`true` | @@ -148,6 +150,7 @@ The following are the properties which can be used to configure Ambari. | kerberos.operation.retry.timeout | The time to wait (in seconds) between failed kerberos operations retries. |`10` | | ldap.sync.username.collision.behavior | Determines how to handle username collision while updating from LDAP.<br/><br/>The following are examples of valid values:<ul><li>`skip`<li>`convert`</ul> |`convert` | | log4j.monitor.delay | Indicates the delay, in milliseconds, for the log4j monitor to check for changes |`300000` | +| logsearch.metadata.cache.expire.timeout | The time, in hours, that the Ambari Server will hold Log File metadata in its internal cache before making a request to the LogSearch Portal to get the latest metadata. |`24` | | logsearch.portal.connect.timeout | The time, in milliseconds, that the Ambari Server will wait while attempting to connect to the LogSearch Portal service. |`5000` | | logsearch.portal.read.timeout | The time, in milliseconds, that the Ambari Server will wait while attempting to read a response from the LogSearch Portal service. |`5000` | | metadata.path | The location on the Ambari Server where the stack resources exist.<br/><br/>The following are examples of valid values:<ul><li>`/var/lib/ambari-server/resources/stacks`</ul> | | @@ -172,7 +175,7 @@ The following are the properties which can be used to configure Ambari. | repo.validation.suffixes.ubuntu | The suffixes to use when validating Ubuntu repositories. |`/dists/%s/Release` | | resources.dir | The location on the Ambari Server where all resources exist, including common services, stacks, and scripts. |`/var/lib/ambari-server/resources/` | | rolling.upgrade.skip.packages.prefixes | A comma-separated list of packages which will be skipped during a stack upgrade. | | -| security.agent.hostname.validate | Determines whether the Ambari Agent host names should be validated against a regular expression to ensure that they are well-formed.<br><br>WARNING: By setting this value to false, host names will not be validated, allowing a possible security vulnerability as described in CVE-2014-3582. See https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Vulnerabilities for more information.|`true` | +| security.agent.hostname.validate | Determines whether the Ambari Agent host names should be validated against a regular expression to ensure that they are well-formed.<br><br>WARNING: By setting this value to false, host names will not be validated, allowing a possible security vulnerability as described in CVE-2014-3582. See https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Vulnerabilities for more information. |`true` | | security.master.key.location | The location on the Ambari Server of the master key file. This is the key to the master keystore. | | | security.master.keystore.location | The location on the Ambari Server of the master keystore file. | | | security.server.cert_name | The name of the file located in the `security.server.keys_dir` directory where certificates will be generated when Ambari uses the `openssl ca` command. |`ca.crt` | @@ -248,9 +251,11 @@ The following are the properties which can be used to configure Ambari. | server.requestlogs.namepattern | The pattern of request log file name |`ambari-access-yyyy_mm_dd.log` | | server.requestlogs.path | The location on the Ambari Server where request logs can be created. | | | server.requestlogs.retaindays | The number of days that request log would be retained. |`15` | +| server.script.threads | The number of threads that should be allocated to run external script. |`4` | | server.script.timeout | The time, in milliseconds, until an external script is killed. |`5000` | | server.stage.command.execution_type | How to execute commands in one stage |`STAGE` | | server.stages.parallel | Determines whether operations in different execution requests can be run concurrently. |`true` | +| server.startup.web.timeout | The time, in seconds, that the ambari-server Python script will wait for Jetty to startup before returning an error code. |`50` | | server.task.timeout | The time, in seconds, before a server-side operation is terminated. |`1200` | | server.timeline.metrics.cache.catchup.interval | The time, in milliseconds, that Ambari Metrics intervals should use when extending the boundaries of the original request.<br/><br/> This property is related to `server.timeline.metrics.cache.disabled`. |`300000` | | server.timeline.metrics.cache.connect.timeout.millis | The time, in milliseconds, to wait while attempting to connect to Ambari Metrics.<br/><br/> This property is related to `server.timeline.metrics.cache.disabled`. |`5000` | http://git-wip-us.apache.org/repos/asf/ambari/blob/76c1fc24/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java b/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java index e1df5bd..eaecf35 100644 --- a/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java +++ b/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java @@ -2639,6 +2639,14 @@ public class Configuration { public static final ConfigurationProperty<Integer> LOGSEARCH_METADATA_CACHE_EXPIRE_TIMEOUT = new ConfigurationProperty<>( "logsearch.metadata.cache.expire.timeout", 24); + /** + * The time, in seconds, that the ambari-server Python script will wait for + * Jetty to startup before returning an error code. + */ + @Markdown(description = "The time, in seconds, that the ambari-server Python script will wait for Jetty to startup before returning an error code.") + public static final ConfigurationProperty<Integer> SERVER_STARTUP_WEB_TIMEOUT = new ConfigurationProperty<>( + "server.startup.web.timeout", 50); + private static final Logger LOG = LoggerFactory.getLogger( Configuration.class); @@ -2696,7 +2704,7 @@ public class Configuration { H2("h2"); private static final Map<String, DatabaseType> m_mappedTypes = - new HashMap<String, Configuration.DatabaseType>(5); + new HashMap<>(5); static { for (DatabaseType databaseType : EnumSet.allOf(DatabaseType.class)) { @@ -2775,12 +2783,12 @@ public class Configuration { public Configuration(Properties properties) { this.properties = properties; - agentConfigsMap = new HashMap<String, String>(); + agentConfigsMap = new HashMap<>(); agentConfigsMap.put(CHECK_REMOTE_MOUNTS.getKey(), getProperty(CHECK_REMOTE_MOUNTS)); agentConfigsMap.put(CHECK_MOUNTS_TIMEOUT.getKey(), getProperty(CHECK_MOUNTS_TIMEOUT)); agentConfigsMap.put(ENABLE_AUTO_AGENT_CACHE_UPDATE.getKey(), getProperty(ENABLE_AUTO_AGENT_CACHE_UPDATE)); - configsMap = new HashMap<String, String>(); + configsMap = new HashMap<>(); configsMap.putAll(agentConfigsMap); configsMap.put(AMBARI_PYTHON_WRAP.getKey(), getProperty(AMBARI_PYTHON_WRAP)); configsMap.put(SRVR_AGENT_HOSTNAME_VALIDATE.getKey(), getProperty(SRVR_AGENT_HOSTNAME_VALIDATE)); @@ -4292,7 +4300,7 @@ public class Configuration { public Map<String, String> getAmbariProperties() { Properties properties = readConfigFile(); - Map<String, String> ambariPropertiesMap = new HashMap<String, String>(); + Map<String, String> ambariPropertiesMap = new HashMap<>(); for(String key : properties.stringPropertyNames()) { ambariPropertiesMap.put(key, properties.getProperty(key)); @@ -5852,7 +5860,7 @@ public class Configuration { // Get and process the configured user type values to convert the comma-delimited string of // user types into a ordered (as found in the comma-delimited value) list of UserType values. String userTypes = getProperty(KERBEROS_AUTH_USER_TYPES); - List<UserType> orderedUserTypes = new ArrayList<UserType>(); + List<UserType> orderedUserTypes = new ArrayList<>(); String[] types = userTypes.split(","); for (String type : types) { http://git-wip-us.apache.org/repos/asf/ambari/blob/76c1fc24/ambari-server/src/main/python/ambari_server/serverConfiguration.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/python/ambari_server/serverConfiguration.py b/ambari-server/src/main/python/ambari_server/serverConfiguration.py index c771b1e..0b4af4f 100644 --- a/ambari-server/src/main/python/ambari_server/serverConfiguration.py +++ b/ambari-server/src/main/python/ambari_server/serverConfiguration.py @@ -181,6 +181,9 @@ VIEWS_DIR_PROPERTY = "views.dir" ACTIVE_INSTANCE_PROPERTY = "active.instance" +# web server startup timeout +WEB_SERVER_STARTUP_TIMEOUT = "server.startup.web.timeout" + #Common setup or upgrade message SETUP_OR_UPGRADE_MSG = "- If this is a new setup, then run the \"ambari-server setup\" command to create the user\n" \ "- If this is an upgrade of an existing setup, run the \"ambari-server upgrade\" command.\n" \ @@ -972,6 +975,25 @@ def remove_password_file(filename): return 0 +def get_web_server_startup_timeout(properties): + """ + Gets the time, in seconds, that the startup script should wait for the web server to bind to + the configured port. If this value is too low, then the startup script will return an + error code even though Ambari is actually starting up. + :param properties: + :return: The timeout value, in seconds. The default is 50. + """ + # get the timeout property and strip it if it exists + timeout = properties[WEB_SERVER_STARTUP_TIMEOUT] + timeout = None if timeout is None else timeout.strip() + + if timeout is None or timeout == "": + timeout = 50 + else: + timeout = int(timeout) + return timeout + + def get_original_master_key(properties, options = None): input = True masterKey = None http://git-wip-us.apache.org/repos/asf/ambari/blob/76c1fc24/ambari-server/src/main/python/ambari_server_main.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/main/python/ambari_server_main.py b/ambari-server/src/main/python/ambari_server_main.py index 66678b0..0cd19cc 100644 --- a/ambari-server/src/main/python/ambari_server_main.py +++ b/ambari-server/src/main/python/ambari_server_main.py @@ -23,22 +23,21 @@ import sys import logging from ambari_commons.exceptions import FatalException -from ambari_commons.logging_utils import get_debug_mode, print_warning_msg, print_info_msg, \ - set_debug_mode_from_options +from ambari_commons.logging_utils import get_debug_mode, print_warning_msg, print_info_msg, set_debug_mode_from_options from ambari_commons.os_check import OSConst from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl -from ambari_commons.os_utils import is_root, run_os_command +from ambari_commons.os_utils import is_root from ambari_server.ambariPath import AmbariPath from ambari_server.dbConfiguration import ensure_dbms_is_running, ensure_jdbc_driver_is_installed from ambari_server.serverConfiguration import configDefaults, find_jdk, get_ambari_properties, \ - get_conf_dir, get_is_persisted, get_is_secure, get_java_exe_path, get_original_master_key, read_ambari_user, \ - get_is_active_instance, update_properties, get_ambari_server_ui_port, \ - PID_NAME, SECURITY_KEY_ENV_VAR_NAME, SECURITY_MASTER_KEY_LOCATION, \ - SETUP_OR_UPGRADE_MSG, check_database_name_property, parse_properties_file, get_missing_properties + get_java_exe_path, read_ambari_user, \ + get_is_active_instance, update_properties, get_ambari_server_ui_port, PID_NAME, \ + check_database_name_property, parse_properties_file, get_missing_properties + +from ambari_server.serverConfiguration import get_web_server_startup_timeout from ambari_server.serverUtils import refresh_stack_hash from ambari_server.setupHttps import get_fqdn -from ambari_server.setupSecurity import generate_env, \ - ensure_can_start_under_current_user +from ambari_server.setupSecurity import generate_env, ensure_can_start_under_current_user from ambari_server.utils import check_reverse_lookup, save_pid, locate_file, locate_all_file_paths, looking_for_pid, \ save_main_pid_ex, check_exitcode, get_live_pids_count, wait_for_ui_start from ambari_server.serverClassPath import ServerClassPath @@ -104,7 +103,6 @@ SERVER_START_CMD_DEBUG_WINDOWS = "{0} " \ SERVER_START_TIMEOUT = 5 #seconds SERVER_START_RETRIES = 4 -WEB_UI_INIT_TIME = 50 #seconds SERVER_PING_TIMEOUT_WINDOWS = 5 SERVER_PING_ATTEMPTS_WINDOWS = 4 @@ -229,9 +227,11 @@ def wait_for_server_start(pidFile, scmStatus): exception = None if server_started: ambari_server_ui_port = get_ambari_server_ui_port(properties) - if not wait_for_ui_start(int(ambari_server_ui_port), WEB_UI_INIT_TIME): + web_server_startup_timeout = get_web_server_startup_timeout(properties) + + if not wait_for_ui_start(int(ambari_server_ui_port), web_server_startup_timeout): exception = FatalException(1, "Server not yet listening on http port " + ambari_server_ui_port + \ - " after " + str(WEB_UI_INIT_TIME) + " seconds. Exiting.") + " after " + str(web_server_startup_timeout) + " seconds. Exiting.") elif get_live_pids_count(pids) <= 0: exitcode = check_exitcode(os.path.join(configDefaults.PID_DIR, EXITCODE_NAME)) exception = FatalException(-1, AMBARI_SERVER_DIE_MSG.format(exitcode, configDefaults.SERVER_OUT_FILE)) http://git-wip-us.apache.org/repos/asf/ambari/blob/76c1fc24/ambari-server/src/test/python/TestAmbariServer.py ---------------------------------------------------------------------- diff --git a/ambari-server/src/test/python/TestAmbariServer.py b/ambari-server/src/test/python/TestAmbariServer.py index 0f704f9..9579c22 100644 --- a/ambari-server/src/test/python/TestAmbariServer.py +++ b/ambari-server/src/test/python/TestAmbariServer.py @@ -4401,8 +4401,8 @@ class TestAmbariServer(TestCase): @patch("os.chown") @patch("ambari_server.setupSecurity.get_master_key_location") @patch("ambari_server.setupSecurity.save_master_key") - @patch("ambari_server_main.get_is_persisted") - @patch("ambari_server_main.get_is_secure") + @patch("ambari_server.setupSecurity.get_is_persisted") + @patch("ambari_server.setupSecurity.get_is_secure") @patch('os.chmod', autospec=True) @patch("ambari_server.serverConfiguration.write_property") @patch("ambari_server.serverConfiguration.get_validated_string_input") @@ -8555,6 +8555,28 @@ class TestAmbariServer(TestCase): self.assertTrue(is_server_runing_method.called) pass + + def test_web_server_startup_timeout(self): + from ambari_server.serverConfiguration import get_web_server_startup_timeout + from ambari_server.serverConfiguration import WEB_SERVER_STARTUP_TIMEOUT + + properties = Properties() + timeout = get_web_server_startup_timeout(properties) + self.assertEquals(50, timeout) + + properties.process_pair(WEB_SERVER_STARTUP_TIMEOUT, "") + timeout = get_web_server_startup_timeout(properties) + self.assertEquals(50, timeout) + + properties.process_pair(WEB_SERVER_STARTUP_TIMEOUT, "120") + timeout = get_web_server_startup_timeout(properties) + self.assertEquals(120, timeout) + + properties.process_pair(WEB_SERVER_STARTUP_TIMEOUT, "120 ") + timeout = get_web_server_startup_timeout(properties) + self.assertEquals(120, timeout) + + def _create_empty_options_mock(self): options = MagicMock() options.ldap_url = None
