I experienced weird classpath issues when deploying LLAP on a from-scratch cluster. Things like, it wasn’t able to find hive-default.xml and had no default configuration provider, so I had to include a hive-default.xml with every possible setting inside the Yarnfile package to get it to run. This doesn’t necessarily apply to HDP but my suggestion would be to trace which classes/configurations are being loaded as much as possible and verify they line up with what you expected.
From: Fernando Antunes <fs.antu...@gmail.com> Sent: Friday, February 28, 2020 2:21 PM To: user@hive.apache.org Subject: LLAP - Hive on Tez - ERROR on SQL query Hi. I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works. When I run a SQL query like select count(*) from database group by column; I've got the following error: Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:492) at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165) The error only happens when I use LLAP, using HiverServer2 works. Anyone knows how to fix this ? More details 0) I’ve searched the Internet and found others with the same error but without answer http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cdb7p194mb0380535868265df6862b849e93...@db7p194mb0380.eurp194.prod.outlook.com%3e https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597 1) Application log form YARN 2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster org.apache.tez.dag.api.TezUncheckedException: java.lang.reflect.InvocationTargetException at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215) at org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184) at org.apache.tez.dag.app.TaskCommunicatorManager.<init>(TaskCommunicatorManager.java:152) at org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163) at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659) at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213) ... 11 more Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:492) at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165) ... 16 more 2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2) From the LlapTaskCommunicator.java file I found : public LlapTaskCommunicator( TaskCommunicatorContext taskCommunicatorContext) { super(taskCommunicatorContext); Credentials credentials = taskCommunicatorContext.getAMCredentials(); if (credentials != null) { @SuppressWarnings("unchecked") Token<LlapTokenIdentifier> llapToken = (Token<LlapTokenIdentifier>)credentials.getToken(LlapTokenIdentifier.KIND_NAME); this.token = llapToken; } else { this.token = null; } if (LOG.isInfoEnabled()) { LOG.info<http://LOG.info>("Task communicator with a token " + token); } Preconditions.checkState((token != null) == UserGroupInformation.isSecurityEnabled()); <=====. Line 165 3) From my application log I could see that token variable is not null Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN; LLAP_TOKEN owner=hive/xxxxx.aaaaa....@xxxxx.aaaaa.com<mailto:owner=hive/xxxxx.aaaaa....@xxxxx.aaaaa.com>, renewer=hive, realUser=, issueDate=1582827878248, maxDate=1584037478248, sequenceNumber=59, masterKeyId=184, cluster hive_llap0, app ID , signing false) 3) Reading Preconditions.java the checkState method * Ensures the truth of an expression involving the state of the calling instance, but not * involving any parameters to the calling method. * * @param expression a boolean expression * @throws IllegalStateException if {@code expression} is false * @see Verify#verify Verify.verify() */ public static void checkState(boolean expression) { if (!expression) { throw new IllegalStateException(); } } 4) from UseGroupInformation.java : the isSecurityEnabled method. ensureInitialized does not change useKerberos variable. public static boolean isSecurityEnabled() { ensureInitialized(); return useKerberos; } 5) useKerberos is true, because my core-site-.xml file has kerberos value set to the HADOOP_SECURITY_AUTHENTICATION property private static synchronized void initialize(Configuration conf, boolean skipRulesSetting) { String value = conf.get(HADOOP_SECURITY_AUTHENTICATION); if (value == null || "simple".equals(value)) { useKerberos = false; } else if ("kerberos".equals(value)) { useKerberos = true; } else { throw new IllegalArgumentException("Invalid attribute value for " + HADOOP_SECURITY_AUTHENTICATION + " of " + value); } 6) Form CommonConfigurationKeys.java, HADOOP_SECURITY_AUTHENTICATION points to hadoop.security.authentication public static final String HADOOP_SECURITY_AUTHENTICATION = "hadoop.security.authentication";