Are you using ambari trunk or ambari 2.0.0 branch builds? Also please confirm: your HDP repos have not changed (I.e. Are you using local repos for the HDP stack packages)?
From: Greg Hill <greg.h...@rackspace.com<mailto:greg.h...@rackspace.com>> Reply-To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>> Date: Wednesday, March 18, 2015 at 12:22 PM To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>> Subject: Did something get broken for webhcat today? Starting this morning, we started seeing this on every single install. I think someone at Hortonworks pushed out a broken RPM or something. Any ideas? This is rather urgent as we are no longer able to provision HDP 2.2 clusters at all because of it. 2015-03-18 15:58:05,982 - Group['hadoop'] {'ignore_failures': False} 2015-03-18 15:58:05,984 - Modifying group hadoop 2015-03-18 15:58:06,080 - Group['nobody'] {'ignore_failures': False} 2015-03-18 15:58:06,081 - Modifying group nobody 2015-03-18 15:58:06,219 - Group['users'] {'ignore_failures': False} 2015-03-18 15:58:06,220 - Modifying group users 2015-03-18 15:58:06,370 - Group['nagios'] {'ignore_failures': False} 2015-03-18 15:58:06,371 - Modifying group nagios 2015-03-18 15:58:06,474 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']} 2015-03-18 15:58:06,475 - Modifying user nobody 2015-03-18 15:58:06,558 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:06,559 - Modifying user hive 2015-03-18 15:58:06,634 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:06,635 - Modifying user mapred 2015-03-18 15:58:06,722 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:06,723 - Modifying user nagios 2015-03-18 15:58:06,841 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-03-18 15:58:06,842 - Modifying user ambari-qa 2015-03-18 15:58:06,963 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:06,964 - Modifying user zookeeper 2015-03-18 15:58:07,093 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-03-18 15:58:07,094 - Modifying user tez 2015-03-18 15:58:07,217 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:07,218 - Modifying user hdfs 2015-03-18 15:58:07,354 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:07,355 - Modifying user yarn 2015-03-18 15:58:07,485 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-03-18 15:58:07,486 - Modifying user hcat 2015-03-18 15:58:07,629 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2015-03-18 15:58:07,631 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} 2015-03-18 15:58:07,768 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if 2015-03-18 15:58:07,769 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True} 2015-03-18 15:58:07,770 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} 2015-03-18 15:58:07,895 - Skipping Link['/etc/hadoop/conf'] due to not_if 2015-03-18 15:58:07,960 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'} 2015-03-18 15:58:08,092 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if': 'test -f /selinux/enforce'} 2015-03-18 15:58:08,240 - Skipping Execute['/bin/echo 0 > /selinux/enforce'] due to only_if 2015-03-18 15:58:08,241 - Directory['/var/log/hadoop'] {'owner': 'root', 'group': 'hadoop', 'mode': 0775, 'recursive': True} 2015-03-18 15:58:08,244 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True} 2015-03-18 15:58:08,250 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True} 2015-03-18 15:58:08,278 - File['/etc/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2015-03-18 15:58:08,288 - File['/etc/hadoop/conf/health_check'] {'content': Template('health_check-v2.j2'), 'owner': 'hdfs'} 2015-03-18 15:58:08,295 - File['/etc/hadoop/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2015-03-18 15:58:08,322 - File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} 2015-03-18 15:58:08,325 - File['/etc/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2015-03-18 15:58:08,330 - File['/etc/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2015-03-18 15:58:09,219 - HdfsDirectory['/user/hcat'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0755, 'owner': 'hcat', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} 2015-03-18 15:58:09,220 - HdfsDirectory['None'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'action': ['create'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'} 2015-03-18 15:58:09,228 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /user/hcat && hadoop --config /etc/hadoop/conf fs -chmod 755 /user/hcat && hadoop --config /etc/hadoop/conf fs -chown hcat /user/hcat'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config /etc/hadoop/conf fs -ls /user/hcat'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} 2015-03-18 15:58:37,822 - Directory['/var/run/webhcat'] {'owner': 'hcat', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2015-03-18 15:58:37,823 - Changing group for /var/run/webhcat from 0 to hadoop 2015-03-18 15:58:37,823 - Directory['/var/log/webhcat'] {'owner': 'hcat', 'group': 'hadoop', 'recursive': True, 'mode': 0755} 2015-03-18 15:58:37,824 - Creating directory Directory['/var/log/webhcat'] 2015-03-18 15:58:37,824 - Changing owner for /var/log/webhcat from 0 to hcat 2015-03-18 15:58:37,824 - Changing group for /var/log/webhcat from 0 to hadoop 2015-03-18 15:58:37,824 - Directory['/etc/hive-webhcat/conf'] {'owner': 'hcat', 'group': 'hadoop', 'recursive': True} 2015-03-18 15:58:37,825 - Changing owner for /etc/hive-webhcat/conf from 0 to hcat 2015-03-18 15:58:37,825 - Changing group for /etc/hive-webhcat/conf from 0 to hadoop 2015-03-18 15:58:37,893 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'} 2015-03-18 15:58:37,896 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive/hive.tar.gz'] {'logoutput': True, 'path': ['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat', 'try_sleep': 0} 2015-03-18 15:58:43,597 - -bash: line 1: 2.2.3.0-2611/hive/hive.tar.gz: No such file or directory 2015-03-18 15:58:43,599 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']} 2015-03-18 15:58:43,601 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} 2015-03-18 15:58:54,904 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'} 2015-03-18 15:58:54,906 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig/pig.tar.gz'] {'logoutput': True, 'path': ['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat', 'try_sleep': 0} 2015-03-18 15:59:00,322 - -bash: line 1: 2.2.3.0-2611/pig/pig.tar.gz: No such file or directory 2015-03-18 15:59:00,323 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']} 2015-03-18 15:59:00,326 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} 2015-03-18 15:59:11,576 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'} 2015-03-18 15:59:11,578 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'path': ['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat', 'try_sleep': 0} 2015-03-18 15:59:17,094 - -bash: line 1: 2.2.3.0-2611/mapreduce/hadoop-streaming.jar: No such file or directory 2015-03-18 15:59:17,097 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']} 2015-03-18 15:59:17,099 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041 2.2.3.0-2611/mapreduce'] {'not_if': '...', 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} 2015-03-18 15:59:28,070 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz 2015-03-18 15:59:28,071 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...} 2015-03-18 15:59:28,090 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml 2015-03-18 15:59:28,091 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2015-03-18 15:59:28,092 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist 2015-03-18 15:59:28,093 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat 2015-03-18 15:59:28,093 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop 2015-03-18 15:59:28,095 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'} 2015-03-18 15:59:28,096 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist 2015-03-18 15:59:28,096 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat 2015-03-18 15:59:28,096 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop 2015-03-18 15:59:28,097 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'} 2015-03-18 15:59:28,179 - Error while executing command 'start': Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat_server.py", line 39, in start webhcat_service(action = 'start') File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat_service.py", line 33, in webhcat_service not_if=no_op_test File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 241, in action_run raise ex Fail: Execution of 'env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start' returned 127. env: /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh: No such file or directory
