AnfengYuan created SPARK-21452: ---------------------------------- Summary: SessionState in HiveClientImpl is never closed Key: SPARK-21452 URL: https://issues.apache.org/jira/browse/SPARK-21452 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0, 2.1.1, 2.1.0, 2.0.2, 2.0.1, 2.0.0 Reporter: AnfengYuan
When I use beeline to connect to spark thriftserver, there are two SessionState created, but when the beeline connection is closed, only one SessionState is closed, the other will never be closed until the spark thriftserver is shutdown. Since SessionState created hdfs and local directories, now there are tens of thousands directories, and I have to delete them manually. One SessionState is created by HiveSession, and is closed by closeSession() in SessionManager, the other one is created by HiveClientImpl, and I can't see any place where it is closed. I want to ask how and where can I close this SessionState? Correct me if I was wrong. ``` 17/07/18 15:34:28.759 INFO ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 17/07/18 15:34:28.772 INFO SessionState: Created local directory: /tmp/288efea6-8cb4-4358-afbd-2663c42bfc6a_resources 17/07/18 15:34:28.774 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/288efea6-8cb4-4358-afbd-2663c42bfc6a 17/07/18 15:34:28.775 INFO SessionState: Created local directory: /tmp/hadoop/288efea6-8cb4-4358-afbd-2663c42bfc6a 17/07/18 15:34:28.777 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/288efea6-8cb4-4358-afbd-2663c42bfc6a/_tmp_space.db 17/07/18 15:34:28.779 INFO HiveSessionImpl: Operation log session directory is created: /tmp/hadoop/operation_logs/288efea6-8cb4-4358-afbd-2663c42bfc6a 17/07/18 15:34:28.862 WARN HiveConf: HiveConf of name hive.mapred.map.tasks.speculative.execution does not exist 17/07/18 15:34:28.862 WARN HiveConf: HiveConf of name hive.server2.thrift.http.max.worker.threads does not exist 17/07/18 15:34:28.868 INFO metastore: Trying to connect to metastore with URI thrift://a01-r15-2bb10-i28-160.jd.local:9083 17/07/18 15:34:28.869 INFO metastore: Connected to metastore. 17/07/18 15:34:28.872 INFO SessionState: Created local directory: /tmp/76fe95c0-4698-4a2e-81a0-3fbf2e863d4d_resources 17/07/18 15:34:28.873 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/76fe95c0-4698-4a2e-81a0-3fbf2e863d4d 17/07/18 15:34:28.874 INFO SessionState: Created local directory: /tmp/hadoop/76fe95c0-4698-4a2e-81a0-3fbf2e863d4d 17/07/18 15:34:28.876 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/76fe95c0-4698-4a2e-81a0-3fbf2e863d4d/_tmp_space.db 17/07/18 15:34:28.877 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is hdfs://ns1/user/hive/warehouse 17/07/18 15:34:28.877 INFO SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=76fe95c0-4698-4a2e-81a0-3fbf2e863d4d, clientType=HIVECLI] 17/07/18 15:34:28.878 INFO metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org