[ https://issues.apache.org/jira/browse/SPARK-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974022#comment-15974022 ]
QQShu1 edited comment on SPARK-20382 at 4/19/17 3:58 AM: --------------------------------------------------------- I find in Hive.java needToCopy HadoopShims.HdfsEncryptionShim srcHdfsEncryptionShim = SessionState.get().getHdfsEncryptionShim(srcFs); in SessionState has a map hdfsEncryptionShims,in HdfsEncryptionShim has a HdfsAdmin, and when we exit beeline ,we call HiveSessionImplwithUGI that FileSystem.closeAllForUGI(sessionUgi). and the fileSystem in HdfsAdmin will be close. Because beeline use one org.apache.hadoop.hive.ql.session.SessionStateļ¼ so the next beeline will get the same object of HdfsAdmin,and in srcHdfsEncryptionShim.isPathEncrypted(srcf), will call the DFSClient.checkOpen, that throw the ava.io.IOException: Filesystem closed. was (Author: qqshu11): I find in Hive.java needToCopy HadoopShims.HdfsEncryptionShim srcHdfsEncryptionShim = SessionState.get().getHdfsEncryptionShim(srcFs); in SessionState has a map hdfsEncryptionShims,in HdfsEncryptionShim has a HdfsAdmin, and when we ext beeline ,we call HiveSessionImplwithUGI that FileSystem.closeAllForUGI(sessionUgi). and the fileSystem in HdfsAdmin will be close. Beacure beeline use one org.apache.hadoop.hive.ql.session.SessionStateļ¼ so the next beeline will get the same object of HdfsAdmin,and in srcHdfsEncryptionShim.isPathEncrypted(srcf), will call the DFSClient.checkOpen, that throw the ava.io.IOException: Filesystem closed. > we meet fileSystem.closed when we run load data on beeline > ---------------------------------------------------------- > > Key: SPARK-20382 > URL: https://issues.apache.org/jira/browse/SPARK-20382 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.0 > Reporter: QQShu1 > > we run on beeline steps as follows: > 1. hadoop fs -put kv1.txt hdfs://spark/user/ > 2.Connect beeline > 3. LOAD DATA INPATH 'hdfs://spark/user/kv1.txt' INTO TABLE src_txt16; > 4.exit beeline > 5.Connect beeline > 6.hadoop fs -put kv1.txt hdfs://spark/user/ > 7.run the LOAD DATA command again , we will meet fileSystem.closed error -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org