Yesha Vora created SPARK-15830:
----------------------------------
Summary: Spark application should get hive tokens only when needed
Key: SPARK-15830
URL: https://issues.apache.org/jira/browse/SPARK-15830
Project: Spark
Issue Type: Improvement
Reporter: Yesha Vora
Currently , All spark application try to get Hive tokens (Even if application
does not use them) if Hive is installed on the cluster.
Due to this practice, spark application which does not require Hive fails when
Hive service (metastore) is down for some reason.
Thus, spark should only try to get Hive tokens when required. It should not
fetch hive token if it is not needed by application.
Example : Spark Pi application does not perform any hive related actions. But
Spark Pi application still fails if hive metastore service is down.
{code}
16/06/08 01:18:42 INFO YarnSparkHadoopUtil: getting token for namenode:
hdfs://xxx:8020/user/xx/.sparkStaging/application_1465347287950_0001
16/06/08 01:18:42 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 7 for xx
on xx.xx.xx.xxx:8020
16/06/08 01:18:43 INFO metastore: Trying to connect to metastore with URI
thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:43 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:43 INFO metastore: Waiting 5 seconds before next connection
attempt.
16/06/08 01:18:48 INFO metastore: Trying to connect to metastore with URI
thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:48 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:48 INFO metastore: Waiting 5 seconds before next connection
attempt.
16/06/08 01:18:53 INFO metastore: Trying to connect to metastore with URI
thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:53 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:53 INFO metastore: Waiting 5 seconds before next connection
attempt.
16/06/08 01:18:59 WARN Hive: Failed to access metastore. This class should not
accessed in runtime.
org.apache.hadoop.hive.ql.metadata.Hive Exception : java.lang.Runtime Exception
: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
