I experienced weird classpath issues when deploying LLAP on a from-scratch
cluster. Things like, it wasn’t able to find hive-default.xml and had no
default configuration provider, so I had to include a hive-default.xml with
every possible setting inside the Yarnfile package to get it to run. This
doesn’t necessarily apply to HDP but my suggestion would be to trace which
classes/configurations are being loaded as much as possible and verify they
line up with what you expected.
From: Fernando Antunes <fs.antu...@gmail.com>
Sent: Friday, February 28, 2020 2:21 PM
To: user@hive.apache.org
Subject: LLAP - Hive on Tez - ERROR on SQL query
Hi.
I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works.
When I run a SQL query like
select count(*) from database group by column;
I've got the following error:
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
The error only happens when I use LLAP, using HiverServer2 works.
Anyone knows how to fix this ?
More details
0) I’ve searched the Internet and found others with the same error but without
answer
http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cdb7p194mb0380535868265df6862b849e93...@db7p194mb0380.eurp194.prod.outlook.com%3e
https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597
1) Application log form YARN
2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting
DAGAppMaster
org.apache.tez.dag.api.TezUncheckedException:
java.lang.reflect.InvocationTargetException
at
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215)
at
org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184)
at
org.apache.tez.dag.app.TaskCommunicatorManager.<init>(TaskCommunicatorManager.java:152)
at
org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163)
at
org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213)
... 11 more
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
... 16 more
2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|:
DAGAppMasterShutdownHook invoked
2) From the LlapTaskCommunicator.java file I found :
public LlapTaskCommunicator(
TaskCommunicatorContext taskCommunicatorContext) {
super(taskCommunicatorContext);
Credentials credentials = taskCommunicatorContext.getAMCredentials();
if (credentials != null) {
@SuppressWarnings("unchecked")
Token<LlapTokenIdentifier> llapToken =
(Token<LlapTokenIdentifier>)credentials.getToken(LlapTokenIdentifier.KIND_NAME);
this.token = llapToken;
} else {
this.token = null;
}
if (LOG.isInfoEnabled()) {
LOG.info<http://LOG.info>("Task communicator with a token " + token);
}
Preconditions.checkState((token != null) ==
UserGroupInformation.isSecurityEnabled()); <=====. Line 165
3) From my application log I could see that token variable is not null
Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN;
LLAP_TOKEN
owner=hive/xxxxx.aaaaa....@xxxxx.aaaaa.com<mailto:owner=hive/xxxxx.aaaaa....@xxxxx.aaaaa.com>,
renewer=hive, realUser=, issueDate=1582827878248, maxDate=1584037478248,
sequenceNumber=59, masterKeyId=184, cluster hive_llap0, app ID , signing false)
3) Reading Preconditions.java the checkState method
* Ensures the truth of an expression involving the state of the calling
instance, but not
* involving any parameters to the calling method.
*
* @param expression a boolean expression
* @throws IllegalStateException if {@code expression} is false
* @see Verify#verify Verify.verify()
*/
public static void checkState(boolean expression) {
if (!expression) {
throw new IllegalStateException();
}
}
4) from UseGroupInformation.java : the isSecurityEnabled method.
ensureInitialized does not change useKerberos variable.
public static boolean isSecurityEnabled() {
ensureInitialized();
return useKerberos;
}
5) useKerberos is true, because my core-site-.xml file has kerberos value set
to the HADOOP_SECURITY_AUTHENTICATION property
private static synchronized void initialize(Configuration conf, boolean
skipRulesSetting) {
String value = conf.get(HADOOP_SECURITY_AUTHENTICATION);
if (value == null || "simple".equals(value)) {
useKerberos = false;
} else if ("kerberos".equals(value)) {
useKerberos = true;
} else {
throw new IllegalArgumentException("Invalid attribute value for " +
HADOOP_SECURITY_AUTHENTICATION +
" of " + value);
}
6) Form CommonConfigurationKeys.java, HADOOP_SECURITY_AUTHENTICATION points to
hadoop.security.authentication
public static final String HADOOP_SECURITY_AUTHENTICATION =
"hadoop.security.authentication";
