Ambari Team, can you please check, why it is required to do manual "kinit" for HiveServer2 startup?
On 4 May 2015 at 22:24, Shaik M <munna.had...@gmail.com> wrote: > Hi, > > After verifying the Amabri agent log, I logged in as "hive" user and I > ran below command. > > /usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs > > after that I have started "hiveserver2" service from Amabri and it started > the service. Now, I can able to connect to beeline client. > > I have re-verified, with destroying the key and the hiveserver2 service > went down after destroying the key. > > Look like Ambari having some issue to initiate the HiveServer2 keytabs. > > -Shaik > > > > > > > > > > On 4 May 2015 at 21:54, Shaik M <munna.had...@gmail.com> wrote: > >> Hi, >> >> >> *I am getting following exception in Hiveserver2 log file:* >> >> 2015-05-04 13:45:23,542 WARN [main]: ipc.Client (Client.java:run(676)) - >> Exception encountered while connecting to the server : >> javax.security.sasl.SaslException: GSS initiate failed [Caused by >> GSSException: No valid credentials provided (Mechanism level: Failed to >> find any Kerberos tgt)] >> 2015-05-04 13:45:23,543 INFO [main]: retry.RetryInvocationHandler >> (RetryInvocationHandler.java:invoke(140)) - Exception while invoking >> getFileInfo of class ClientNamenodeProtocolTranslatorPB over >> sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020 after 8 fail over >> attempts. Trying to fail over immediately. >> java.io.IOException: Failed on local exception: java.io.IOException: >> javax.security.sasl.SaslException: GSS initiate failed [Caused by >> GSSException: No valid credentials provided (Mechanism level: Failed to >> find any Kerberos tgt)]; >> >> Thanks, >> Shaik >> >> (If it is not right place to post this query, please help me to know the >> correct group) >> >> On 4 May 2015 at 18:55, Shaik M <munna.had...@gmail.com> wrote: >> >>> Hi, >>> >>> I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this >>> cluster. >>> >>> After enabling the security Ambari unable start the Hive server 2. >>> >>> I have verified keytabs and it's working fine. Please find the below >>> ambari string process log below. not shown any output in standerr window. >>> >>> please let me know the how to resolve this issue. >>> >>> Thanks, >>> Shaik M >>> >>> stdout: /var/lib/ambari-agent/data/output-1832.txt >>> >>> 2015-05-04 10:47:25,616 - Execute['mkdir -p >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" >>> --retry 10 >>> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip >>> -o >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >>> {'environment': ..., 'not_if': 'test -e >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', >>> 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']} >>> 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" >>> --retry 10 >>> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip >>> -o >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >>> due to not_if >>> 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f >>> US_export_policy.jar; unzip -o -j -q >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >>> {'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e >>> /usr/jdk64/jdk1.7.0_67/jre/lib/security && test -f >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', >>> 'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'} >>> 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False} >>> 2015-05-04 10:47:25,720 - Modifying group hadoop >>> 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False} >>> 2015-05-04 10:47:25,785 - Modifying group nobody >>> 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False} >>> 2015-05-04 10:47:25,840 - Modifying group users >>> 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False} >>> 2015-05-04 10:47:25,887 - Modifying group nagios >>> 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'nobody']} >>> 2015-05-04 10:47:25,937 - Modifying user nobody >>> 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'users']} >>> 2015-05-04 10:47:26,013 - Modifying user oozie >>> 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': >>> False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,064 - Modifying user hive >>> 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,099 - Modifying user mapred >>> 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', >>> 'ignore_failures': False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,132 - Modifying user nagios >>> 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'users']} >>> 2015-05-04 10:47:26,166 - Modifying user ambari-qa >>> 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,199 - Modifying user zookeeper >>> 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': >>> False, 'groups': [u'users']} >>> 2015-05-04 10:47:26,233 - Modifying user tez >>> 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': >>> False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,266 - Modifying user hdfs >>> 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', >>> 'ignore_failures': False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,299 - Modifying user sqoop >>> 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': >>> False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,332 - Modifying user hcat >>> 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': >>> False, 'groups': [u'hadoop']} >>> 2015-05-04 10:47:26,366 - Modifying user yarn >>> 2015-05-04 10:47:26,399 - >>> File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': >>> StaticFile('changeToSecureUid.sh'), 'mode': 0555} >>> 2015-05-04 10:47:26,401 - >>> Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa >>> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa >>> 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} >>> 2015-05-04 10:47:26,431 - Skipping >>> Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa >>> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa >>> 2>/dev/null'] due to not_if >>> 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': >>> 'root', 'group': 'root', 'recursive': True} >>> 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls >>> /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} >>> 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if >>> 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] >>> {'content': InlineTemplate(...), 'owner': 'root'} >>> 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] >>> {'only_if': 'test -f /selinux/enforce'} >>> 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', >>> 'group': 'hadoop', 'mode': 0775, 'recursive': True} >>> 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', >>> 'group': 'root', 'recursive': True} >>> 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', >>> 'recursive': True} >>> 2015-05-04 10:47:26,590 - >>> File['/etc/hadoop/conf/commons-logging.properties'] {'content': >>> Template('commons-logging.properties.j2'), 'owner': 'root'} >>> 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': >>> Template('health_check-v2.j2'), 'owner': 'root'} >>> 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] >>> {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} >>> 2015-05-04 10:47:26,608 - >>> File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content': >>> Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} >>> 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] >>> {'content': StaticFile('task-log4j.properties'), 'mode': 0755} >>> 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` >>> >/dev/null 2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls >>> /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps `cat >>> /var/run/hive/hive-server.pid` >/dev/null 2>&1)'} >>> 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] >>> {'security_enabled': True, 'keytab': >>> '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >>> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >>> '/usr/bin/kinit', 'mode': 0777, 'owner': 'hive', 'bin_dir': >>> '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} >>> 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': >>> True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >>> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >>> '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': >>> '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} >>> 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, >>> 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >>> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >>> '/usr/bin/kinit', 'action': ['create'], 'bin_dir': >>> '/usr/hdp/current/hadoop-client/bin'} >>> 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt >>> /etc/security/keytabs/hdfs.headless.keytab hdfs'] {'user': 'hdfs'} >>> 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs >>> -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` >>> /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs >>> -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs >>> -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive >>> /apps/hive/warehouse /user/hive'] {'not_if': "su - hdfs -c 'export >>> PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config >>> /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs', >>> 'path': ['/usr/hdp/current/hadoop-client/bin']} >>> 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config >>> /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo >>> "-p"` /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf >>> fs -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs >>> -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive >>> /apps/hive/warehouse /user/hive'] due to not_if >>> 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': >>> 'hive', 'group': 'hadoop', 'recursive': True} >>> 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', >>> 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, >>> 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} >>> 2015-05-04 10:47:31,168 - Generating config: >>> /etc/hive/conf.server/mapred-site.xml >>> 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] >>> {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', >>> 'mode': 0644, 'encoding': 'UTF-8'} >>> 2015-05-04 10:47:31,173 - Writing >>> File['/etc/hive/conf.server/mapred-site.xml'] because contents don't match >>> 2015-05-04 10:47:31,174 - >>> File['/etc/hive/conf.server/hive-default.xml.template'] {'owner': 'hive', >>> 'group': 'hadoop'} >>> 2015-05-04 10:47:31,175 - >>> File['/etc/hive/conf.server/hive-env.sh.template'] {'owner': 'hive', >>> 'group': 'hadoop'} >>> 2015-05-04 10:47:31,177 - >>> File['/etc/hive/conf.server/hive-exec-log4j.properties'] {'content': '...', >>> 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >>> 2015-05-04 10:47:31,178 - >>> File['/etc/hive/conf.server/hive-log4j.properties'] {'content': '...', >>> 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >>> 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', >>> 'group': 'hadoop', 'recursive': True} >>> 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', >>> 'conf_dir': '/etc/hive/conf', 'mode': 0644, 'configuration_attributes': >>> ..., 'owner': 'hive', 'configurations': ...} >>> 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml >>> 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': >>> 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, >>> 'encoding': 'UTF-8'} >>> 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] >>> because contents don't match >>> 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] >>> {'owner': 'hive', 'group': 'hadoop'} >>> 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] >>> {'owner': 'hive', 'group': 'hadoop'} >>> 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] >>> {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >>> 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] >>> {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >>> 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', >>> 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, >>> 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} >>> 2015-05-04 10:47:31,236 - Generating config: >>> /etc/hive/conf.server/hive-site.xml >>> 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] >>> {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', >>> 'mode': 0644, 'encoding': 'UTF-8'} >>> 2015-05-04 10:47:31,245 - Writing >>> File['/etc/hive/conf.server/hive-site.xml'] because contents don't match >>> 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] >>> {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop'} >>> 2015-05-04 10:47:31,253 - Execute['hive mkdir -p >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f >>> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp >>> /usr/share/java/mysql-connector-java.jar >>> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': >>> ..., 'path': ['/bin', '/usr/bin/'], 'creates': >>> '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if': >>> 'test -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'} >>> 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p >>> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f >>> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp >>> /usr/share/java/mysql-connector-java.jar >>> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if >>> 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ && >>> curl -kf -x "" --retry 5 >>> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar >>> -o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f >>> DBConnectionVerification.jar]'} >>> 2015-05-04 10:47:31,370 - >>> File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script'] {'content': >>> Template('startHiveserver2.sh.j2'), 'mode': 0755} >>> 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', >>> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >>> 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', >>> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >>> 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', >>> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >>> 2015-05-04 10:47:31,453 - Could not verify HDP version by calling >>> '/usr/bin/hdp-select versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: >>> 2.2.4.2-2 >>> . >>> 2015-05-04 10:47:31,531 - Could not verify HDP version by calling >>> '/usr/bin/hdp-select versions > /tmp/tmpHFa97f'. Return Code: 0, Output: >>> 2.2.4.2-2 >>> . >>> 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 >>> /var/lib/ambari-agent/data/tmp/start_hiveserver2_script >>> /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log >>> /var/run/hive/hive-server.pid /etc/hive/conf.server /var/log/hive'] >>> {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null >>> 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1', 'user': >>> 'hive', 'path': >>> ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']} >>> 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp >>> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar >>> org.apache.ambari.server.DBConnectionVerification >>> 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true' >>> hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': >>> ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': >>> 10} >>> >>> >> >
