RE: LLAP - Hive on Tez - ERROR on SQL query

2020-02-28 Thread Aaron Grubb 
I experienced weird classpath issues when deploying LLAP on a from-scratch 
cluster. Things like, it wasn’t able to find hive-default.xml and had no 
default configuration provider, so I had to include a hive-default.xml with 
every possible setting inside the Yarnfile package to get it to run. This 
doesn’t necessarily apply to HDP but my suggestion would be to trace which 
classes/configurations are being loaded as much as possible and verify they 
line up with what you expected.

From: Fernando Antunes 
Sent: Friday, February 28, 2020 2:21 PM
To: user@hive.apache.org
Subject: LLAP - Hive on Tez - ERROR on SQL query

Hi.

I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works.

When I run a SQL query like

 select count(*) from database group by column;

I've got the following error:

Caused by: java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at 
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.(LlapTaskCommunicator.java:165)

The error only happens when I use LLAP, using HiverServer2 works.



Anyone knows how to fix this ?




More details

0) I’ve searched the Internet and found others with the same error but without 
answer

http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cdb7p194mb0380535868265df6862b849e93...@db7p194mb0380.eurp194.prod.outlook.com%3e
https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597


1) Application log form YARN


2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting 
DAGAppMaster
org.apache.tez.dag.api.TezUncheckedException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.(TaskCommunicatorManager.java:152)
at 
org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163)
at 
org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213)
... 11 more
Caused by: java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at 
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.(LlapTaskCommunicator.java:165)
... 16 more
2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: 
DAGAppMasterShutdownHook invoked

2) From the  LlapTaskCommunicator.java file I found :


 public LlapTaskCommunicator(
  TaskCommunicatorContext taskCommunicatorContext) {
super(taskCommunicatorContext);
Credentials credentials = taskCommunicatorContext.getAMCredentials();
if (credentials != null) {
  @SuppressWarnings("unchecked")
  Token llapToken =
  
(Token)credentials.getToken(LlapTokenIdentifier.KIND_NAME);
  this.token = llapToken;
} else {
  this.token = null;
}
if (LOG.isInfoEnabled()) {
  LOG.info<http://LOG.info>("Task communicator with a token " + token);
}
Preconditions.checkState((token != null) == 
UserGroupInformation.isSecurityEnabled()); <=. Line 165

3) From my application log I could see that token variable is not null

Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN; 
LLAP_TOKEN 
owner=hive/x.a@x.a.com<mailto:owner=hive/x.a@x.a.com>,
 renewer=hive, realUser=, issueDate=1582827878248, maxDate=1584037478248, 
sequenceNumber=59, masterKeyId=184, cluster hive_llap0, app ID , signing false)


3) Reading Preco

LLAP - Hive on Tez - ERROR on SQL query

2020-02-28 Thread Fernando Antunes 
Hi. 

I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled 
works. 

When I run a SQL query like 

 select count(*) from database group by column;

I've got the following error:

Caused by: java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at 
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.(LlapTaskCommunicator.java:165)

The error only happens when I use LLAP, using HiverServer2 works.


Anyone knows how to fix this ?




More details

0) I’ve searched the Internet and found others with the same error but without 
answer

http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cdb7p194mb0380535868265df6862b849e93...@db7p194mb0380.eurp194.prod.outlook.com%3e
 

https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597
 



1) Application log form YARN


2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting 
DAGAppMaster
org.apache.tez.dag.api.TezUncheckedException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.(TaskCommunicatorManager.java:152)
at 
org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163)
at 
org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213)
... 11 more
Caused by: java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at 
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.(LlapTaskCommunicator.java:165)
... 16 more
2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: 
DAGAppMasterShutdownHook invoked

2) From the  LlapTaskCommunicator.java file I found :

 public LlapTaskCommunicator(
  TaskCommunicatorContext taskCommunicatorContext) {
super(taskCommunicatorContext);
Credentials credentials = taskCommunicatorContext.getAMCredentials();
if (credentials != null) {
  @SuppressWarnings("unchecked")
  Token llapToken =
  
(Token)credentials.getToken(LlapTokenIdentifier.KIND_NAME);
  this.token = llapToken;
} else {
  this.token = null;
}
if (LOG.isInfoEnabled()) {
  LOG.info("Task communicator with a token " + token);
}
Preconditions.checkState((token != null) == 
UserGroupInformation.isSecurityEnabled()); <=. Line 165

3) From my application log I could see that token variable is not null 

Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN; 
LLAP_TOKEN owner=hive/x.a@x.a.com, renewer=hive, realUser=, 
issueDate=1582827878248, maxDate=1584037478248, sequenceNumber=59, 
masterKeyId=184, cluster hive_llap0, app ID , signing false)


3) Reading Preconditions.java the checkState method


   * Ensures the truth of an expression involving the state of the calling 
instance, but not
   * involving any parameters to the calling method.
   *
   * @param expression a boolean expression
   * @throws IllegalStateException if {@code expression} is false
   * @see Verify#verify Verify.verify()
   */
  public static void checkState(boolean expression) {
if (!expression) {
  throw new IllegalStateException();
}
  }

4) from UseGroupInformation.java : the isSecurityEnabled method.  
ensureInitialized does not change