[
https://issues.apache.org/jira/browse/AMBARI-15342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yusaku Sako updated AMBARI-15342:
---------------------------------
Priority: Critical (was: Major)
> AMS Grafana start failed with permission denied error on changing user
> ----------------------------------------------------------------------
>
> Key: AMBARI-15342
> URL: https://issues.apache.org/jira/browse/AMBARI-15342
> Project: Ambari
> Issue Type: Bug
> Components: ambari-metrics
> Affects Versions: 2.2.2
> Reporter: Siddharth Wagle
> Assignee: Siddharth Wagle
> Priority: Critical
> Fix For: 2.2.2
>
> Attachments: AMBARI-15342.patch
>
>
> Grafana service failed to start when tried to start from the Ambari UI [from
> Ambari Metrics service page, select drop down "Start"].
> Failure message from logs : */var/log/ambari-metrics-grafana/grafana.out:
> Permission denied\nFAILED"*
> Complete logs:
> {noformat}
> {
> "href" :
> "http://172.22.110.160:8080/api/v1/clusters/cl1/requests/4/tasks/181";,
> "Tasks" : {
> "attempt_cnt" : 1,
> "cluster_name" : "cl1",
> "command" : "START",
> "command_detail" : "METRICS_GRAFANA START",
> "end_time" : 1457416332931,
> "error_log" : "/var/lib/ambari-agent/data/errors-181.txt",
> "exit_code" : 1,
> "host_name" : "os-r6-dggzcu-ambari-rare-19-5.novalocal",
> "id" : 181,
> "output_log" : "/var/lib/ambari-agent/data/output-181.txt",
> "request_id" : 4,
> "role" : "METRICS_GRAFANA",
> "stage_id" : 4,
> "start_time" : 1457416296331,
> "status" : "FAILED",
> "stderr" : "Traceback (most recent call last):\n File
> \"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana.py\",
> line 70, in <module>\n AmsGrafana().execute()\n File
> \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
> line 219, in execute\n method(env)\n File
> \"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana.py\",
> line 48, in start\n user=params.ams_user\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line
> 154, in __init__\n self.env.run()\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
> line 158, in run\n self.run_action(resource, action)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\",
> line 121, in run_action\n provider_action()\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\",
> line 238, in action_run\n tries=self.resource.tries,
> try_sleep=self.resource.try_sleep)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 70, in inner\n result = function(command, **kwargs)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 92, in checked_call\n tries=tries, try_sleep=try_sleep)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 140, in _call_wrapper\n result = _call(command, **kwargs_copy)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 291, in _call\n raise
> Fail(err_msg)\nresource_management.core.exceptions.Fail: Execution of
> '/usr/sbin/ambari-metrics-grafana start' returned 1. ######## Hortonworks
> #############\nThis is MOTD message, added for testing in qe infra\nStarting
> Ambari Metrics Grafana: .... /usr/sbin/ambari-metrics-grafana: line 114:
> /var/log/ambari-metrics-grafana/grafana.out: Permission denied\nFAILED",
> "stdout" : "2016-03-08 05:51:46,460 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169\n2016-03-08 05:51:46,460 - Checking if need to create
> versioned conf dir /etc/hadoop/2.4.0.0-169/0\n2016-03-08 05:51:46,467 -
> call['conf-select create-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False, 'stderr': -1}\n2016-03-08 05:51:46,602 - call returned (1,
> '/etc/hadoop/2.4.0.0-169/0 exist already', '')\n2016-03-08 05:51:46,602 -
> checked_call['conf-select set-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False}\n2016-03-08 05:51:46,788 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')\n2016-03-08
> 05:51:46,788 - Ensuring that hadoop has the correct symlink
> structure\n2016-03-08 05:51:46,789 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf\n2016-03-08 05:51:47,321 - The hadoop
> conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on
> it for version 2.4.0.0-169\n2016-03-08 05:51:47,322 - Checking if need to
> create versioned conf dir /etc/hadoop/2.4.0.0-169/0\n2016-03-08 05:51:47,322
> - call['conf-select create-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False, 'stderr': -1}\n2016-03-08 05:51:47,386 - call returned (1,
> '/etc/hadoop/2.4.0.0-169/0 exist already', '')\n2016-03-08 05:51:47,387 -
> checked_call['conf-select set-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False}\n2016-03-08 05:51:47,439 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')\n2016-03-08
> 05:51:47,440 - Ensuring that hadoop has the correct symlink
> structure\n2016-03-08 05:51:47,440 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf\n2016-03-08 05:51:47,442 -
> Group['cstm-knox-group'] {}\n2016-03-08 05:51:47,448 - Group['hadoop']
> {}\n2016-03-08 05:51:47,449 - Group['users'] {}\n2016-03-08 05:51:47,449 -
> Group['cstm-spark'] {}\n2016-03-08 05:51:47,449 - User['atlas'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08
> 05:51:47,451 - User['cstm-hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups':
> True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,453 - User['cstm-sqoop']
> {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups':
> ['hadoop']}\n2016-03-08 05:51:47,454 - User['cstm-ams'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,455
> - User['cstm-tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups':
> ['users']}\n2016-03-08 05:51:47,456 - User['cstm-storm'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,457
> - User['cstm-knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True,
> 'groups': ['hadoop']}\n2016-03-08 05:51:47,458 - User['cstm-flume'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08
> 05:51:47,459 - User['cstm-kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups':
> True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,460 - User['cstm-hcat']
> {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups':
> ['hadoop']}\n2016-03-08 05:51:47,462 - User['cstm-mahout'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,463
> - User['cstm-hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True,
> 'groups': ['hadoop']}\n2016-03-08 05:51:47,464 - User['cstm-hdfs'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08
> 05:51:47,465 - User['cstm-falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups':
> True, 'groups': ['users']}\n2016-03-08 05:51:47,466 - User['cstm-accumulo']
> {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups':
> ['hadoop']}\n2016-03-08 05:51:47,467 - User['ambari-qa'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}\n2016-03-08 05:51:47,468
> - User['cstm-zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True,
> 'groups': ['hadoop']}\n2016-03-08 05:51:47,470 - User['cstm-oozie'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}\n2016-03-08
> 05:51:47,471 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True,
> 'groups': ['hadoop']}\n2016-03-08 05:51:47,472 - User['mapred'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08
> 05:51:47,473 - User['cstm-spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups':
> True, 'groups': ['hadoop']}\n2016-03-08 05:51:47,474 - User['hcat'] {'gid':
> 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}\n2016-03-08
> 05:51:47,475 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content':
> StaticFile('changeToSecureUid.sh'), 'mode': 0555}\n2016-03-08 05:51:47,690 -
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}\n2016-03-08
> 05:51:47,715 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh
> ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> due to not_if\n2016-03-08 05:51:47,716 - Directory['/tmp/hbase-hbase']
> {'owner': 'cstm-hbase', 'recursive': True, 'mode': 0775, 'cd_access':
> 'a'}\n2016-03-08 05:51:48,114 -
> File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content':
> StaticFile('changeToSecureUid.sh'), 'mode': 0555}\n2016-03-08 05:51:48,416 -
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh cstm-hbase
> /home/cstm-hbase,/tmp/cstm-hbase,/usr/bin/cstm-hbase,/var/log/cstm-hbase,/tmp/hbase-hbase']
> {'not_if': '(test $(id -u cstm-hbase) -gt 1000) || (false)'}\n2016-03-08
> 05:51:48,427 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh
> cstm-hbase
> /home/cstm-hbase,/tmp/cstm-hbase,/usr/bin/cstm-hbase,/var/log/cstm-hbase,/tmp/hbase-hbase']
> due to not_if\n2016-03-08 05:51:48,428 - Group['cstm-hdfs'] {}\n2016-03-08
> 05:51:48,428 - User['cstm-hdfs'] {'fetch_nonlocal_groups': True, 'groups':
> ['hadoop', 'cstm-hdfs']}\n2016-03-08 05:51:48,429 - Directory['/etc/hadoop']
> {'mode': 0755}\n2016-03-08 05:51:48,577 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content':
> InlineTemplate(...), 'owner': 'cstm-hdfs', 'group': 'hadoop'}\n2016-03-08
> 05:51:48,683 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir']
> {'owner': 'cstm-hdfs', 'group': 'hadoop', 'mode': 0777}\n2016-03-08
> 05:51:48,895 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce )
> || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True,
> 'only_if': 'test -f /selinux/enforce'}\n2016-03-08 05:51:49,041 -
> Directory['/grid/0/log/hdfs'] {'owner': 'root', 'mode': 0775, 'group':
> 'hadoop', 'recursive': True, 'cd_access': 'a'}\n2016-03-08 05:51:49,639 -
> Directory['/grid/0/pid/hdfs'] {'owner': 'root', 'group': 'root', 'recursive':
> True, 'cd_access': 'a'}\n2016-03-08 05:51:49,974 -
> Directory['/tmp/hadoop-cstm-hdfs'] {'owner': 'cstm-hdfs', 'recursive': True,
> 'cd_access': 'a'}\n2016-03-08 05:51:50,119 -
> File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties']
> {'content': Template('commons-logging.properties.j2'), 'owner':
> 'cstm-hdfs'}\n2016-03-08 05:51:50,225 -
> File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content':
> Template('health_check.j2'), 'owner': 'cstm-hdfs'}\n2016-03-08 05:51:50,334 -
> File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ...,
> 'owner': 'cstm-hdfs', 'group': 'hadoop', 'mode': 0644}\n2016-03-08
> 05:51:50,504 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties']
> {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'cstm-hdfs',
> 'group': 'hadoop'}\n2016-03-08 05:51:50,635 -
> File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content':
> StaticFile('task-log4j.properties'), 'mode': 0755}\n2016-03-08 05:51:50,766 -
> File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner':
> 'cstm-hdfs', 'group': 'hadoop'}\n2016-03-08 05:51:50,874 -
> File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'cstm-hdfs',
> 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d
> /etc/hadoop/conf', 'group': 'hadoop'}\n2016-03-08 05:51:51,039 -
> File['/etc/hadoop/conf/topology_script.py'] {'content':
> StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf',
> 'mode': 0755}\n2016-03-08 05:51:52,004 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169\n2016-03-08 05:51:52,004 - Checking if need to create
> versioned conf dir /etc/hadoop/2.4.0.0-169/0\n2016-03-08 05:51:52,004 -
> call['conf-select create-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False, 'stderr': -1}\n2016-03-08 05:51:52,056 - call returned (1,
> '/etc/hadoop/2.4.0.0-169/0 exist already', '')\n2016-03-08 05:51:52,057 -
> checked_call['conf-select set-conf-dir --package hadoop --stack-version
> 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet':
> False}\n2016-03-08 05:51:52,112 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')\n2016-03-08
> 05:51:52,112 - Ensuring that hadoop has the correct symlink
> structure\n2016-03-08 05:51:52,112 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf\n2016-03-08 05:51:52,138 -
> Directory['/etc/ambari-metrics-grafana/conf'] {'owner': 'cstm-ams', 'group':
> 'hadoop', 'recursive': True, 'mode': 0755}\n2016-03-08 05:51:52,206 -
> Changing owner for /etc/ambari-metrics-grafana/conf from 2554 to
> cstm-ams\n2016-03-08 05:51:52,206 - Changing group for
> /etc/ambari-metrics-grafana/conf from 2551 to hadoop\n2016-03-08 05:51:52,265
> - Directory['/var/log/ambari-metrics-grafana'] {'owner': 'cstm-ams', 'group':
> 'hadoop', 'recursive': True, 'mode': 0755}\n2016-03-08 05:51:52,322 -
> Changing owner for /var/log/ambari-metrics-grafana from 0 to
> cstm-ams\n2016-03-08 05:51:52,322 - Changing group for
> /var/log/ambari-metrics-grafana from 0 to hadoop\n2016-03-08 05:51:52,368 -
> Directory['/var/lib/ambari-metrics-grafana'] {'owner': 'cstm-ams', 'group':
> 'hadoop', 'recursive': True, 'mode': 0755}\n2016-03-08 05:51:52,426 -
> Changing owner for /var/lib/ambari-metrics-grafana from 0 to
> cstm-ams\n2016-03-08 05:51:52,426 - Changing group for
> /var/lib/ambari-metrics-grafana from 0 to hadoop\n2016-03-08 05:51:52,477 -
> Directory['/var/run/ambari-metrics-grafana'] {'owner': 'cstm-ams', 'group':
> 'hadoop', 'recursive': True, 'mode': 0755}\n2016-03-08 05:51:52,609 -
> Changing owner for /var/run/ambari-metrics-grafana from 0 to
> cstm-ams\n2016-03-08 05:51:52,610 - Changing group for
> /var/run/ambari-metrics-grafana from 0 to hadoop\n2016-03-08 05:51:52,710 -
> File['/etc/ambari-metrics-grafana/conf/ams-grafana-env.sh'] {'content':
> InlineTemplate(...), 'owner': 'cstm-ams', 'group': 'hadoop'}\n2016-03-08
> 05:51:52,916 - Writing
> File['/etc/ambari-metrics-grafana/conf/ams-grafana-env.sh'] because contents
> don't match\n2016-03-08 05:51:53,023 - Changing owner for
> /etc/ambari-metrics-grafana/conf/ams-grafana-env.sh from 0 to
> cstm-ams\n2016-03-08 05:51:53,023 - Changing group for
> /etc/ambari-metrics-grafana/conf/ams-grafana-env.sh from 0 to
> hadoop\n2016-03-08 05:51:53,051 -
> File['/etc/ambari-metrics-grafana/conf/ams-grafana.ini'] {'content':
> InlineTemplate(...), 'owner': 'cstm-ams', 'group': 'hadoop'}\n2016-03-08
> 05:51:53,225 - Writing
> File['/etc/ambari-metrics-grafana/conf/ams-grafana.ini'] because contents
> don't match\n2016-03-08 05:51:53,285 - Changing owner for
> /etc/ambari-metrics-grafana/conf/ams-grafana.ini from 0 to
> cstm-ams\n2016-03-08 05:51:53,285 - Changing group for
> /etc/ambari-metrics-grafana/conf/ams-grafana.ini from 0 to hadoop\n2016-03-08
> 05:51:53,306 - Execute['/usr/sbin/ambari-metrics-grafana stop'] {'user':
> 'cstm-ams'}\n2016-03-08 05:51:58,474 -
> Execute['/usr/sbin/ambari-metrics-grafana start'] {'user': 'cstm-ams'}",
> "structured_out" : { }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
