[
https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kent Yao resolved SPARK-34852.
------------------------------
Fix Version/s: 3.2.0
Assignee: ulysses you
Resolution: Fixed
> Close Hive session state should use withHiveState
> -------------------------------------------------
>
> Key: SPARK-34852
> URL: https://issues.apache.org/jira/browse/SPARK-34852
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: ulysses you
> Assignee: ulysses you
> Priority: Minor
> Fix For: 3.2.0
>
>
> Some reason:
> 1. Shutdown hook is invoked using different thread.
> 2. Hive may use metasotre client again during closing
> Otherwise, we may get such expcetion with custom hive metastore version
> {code:java}
> 21/03/24 13:26:18 INFO session.SessionState: Failed to remove classloaders
> from DataNucleus
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
> at
> org.apache.hadoop.hive.ql.session.SessionState.unCacheDataNucleusClassLoaders(SessionState.java:1546)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1536)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.closeState(HiveClientImpl.scala:172)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$new$1(HiveClientImpl.scala:175)
> at
> org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
> at
> org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
