[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saisai Shao reassigned SPARK-22290: ----------------------------------- Assignee: Marcelo Vanzin > Starting second context in same JVM fails to get new Hive delegation token > -------------------------------------------------------------------------- > > Key: SPARK-22290 > URL: https://issues.apache.org/jira/browse/SPARK-22290 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Marcelo Vanzin > Assignee: Marcelo Vanzin > Fix For: 2.3.0 > > > Consider the following pyspark script: > {code} > sc = SparkContext() > // do stuff > sc.stop() > // do some other stuff > sc = SparkContext() > {code} > That code didn't use to work at all in 2.2 (failure to create the second > context), but makes more progress in 2.3. But it fails to create new Hive > delegation tokens; you see this error in the output: > {noformat} > 17/10/16 16:26:50 INFO security.HadoopFSDelegationTokenProvider: getting > token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1714191595_19, > ugi=blah(auth:KERBEROS)]] > 17/10/16 16:26:50 INFO hive.metastore: Trying to connect to metastore with > URI blah > 17/10/16 16:26:50 INFO hive.metastore: Connected to metastore. > 17/10/16 16:26:50 ERROR metadata.Hive: MetaException(message:Delegation Token > can be issued only with kerberos authentication. Current > AuthenticationMethod: TOKEN) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result.read(ThriftHiveMetastore > {noformat} > The error is printed in the logs but it doesn't cause the app to fail (which > might be considered wrong). > The effect is that when that old delegation token expires the new app will > fail. > But the real issue here is that Spark shouldn't be mixing delegation tokens > from different apps. It should try harder to isolate a set of delegation > tokens to a single app submission. > And, in the case of Hive, there are many situations where a delegation token > isn't needed at all. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org