xianyinxin opened a new pull request #25197: [SPARK-21067][SQL] Initialize the HdfsEncryptionShim and the underlying FileSystem(FS) e… URL: https://github.com/apache/spark/pull/25197 ## What changes were proposed in this pull request? This pr fix the "Filesystem Closed" exception that caused by the closing of a hive session in thriftserver. A `sessionState` holds a `hdfsEncryptionShim`, which holds a `fileSystem`. When a session was closed, the fileSystem may be closed. While the fileSystem is cached, so the subsequent using of the same fileSystem may throw "Filesystem closed" exception if previous session closes the fileSystem. We can move the "holding of the `fileSystem`" from session to thriftserver by triggering `SessionState#getHdfsEncryptionShim` eagerly at the starting phase of the thriftserver, see below, ``` HiveThriftServer2#main() hiveClientImpl val state = newState SessionState.start(state) SessionState.setCurrentSession(state) **state.getHdfsEncryptionShim** ``` In hive session, the new sessionState will reuse the `fileSystem` created by thriftserver, ``` CliService openSession HiveSessionImpl init sessionState = new SessionState SessionState.setCurrentSession(sessionState) executeStatementInternal SessionState.setCurrentSession(sessionState) operation.run() hiveSession.getSessionManager().submitBackgroundOperation(operation) **// If some code calls "state.getHdfsEncryptionShim", the cached `fileSystem` is reused.** ``` ## How was this patch tested? manual test
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
