[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-34852: ------------------------------------ Assignee: (was: Apache Spark) > Close Hive session state should use withHiveState > ------------------------------------------------- > > Key: SPARK-34852 > URL: https://issues.apache.org/jira/browse/SPARK-34852 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: ulysses you > Priority: Minor > > Some reason: > 1. Shutdown hook is invoked using different thread. > 2. Hive may use metasotre client again during closing > Otherwise, we may get such expcetion with custom hive metastore version > {code:java} > 21/03/24 13:26:18 INFO session.SessionState: Failed to remove classloaders > from DataNucleus > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386) > at > org.apache.hadoop.hive.ql.session.SessionState.unCacheDataNucleusClassLoaders(SessionState.java:1546) > at > org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1536) > at > org.apache.spark.sql.hive.client.HiveClientImpl.closeState(HiveClientImpl.scala:172) > at > org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$new$1(HiveClientImpl.scala:175) > at > org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214) > at > org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org