Thank you I've been researching based on your opinions, and found the below two solutions. These are the answers for who has FileSystem.closed issue like me.
- close it in your cleanup method and you have JVM reuse turned on (mapred.job.reuse.jvm.num.tasks) - set "fs.hdfs.impl.disable,cache' to turn in the conf, and new instances don't get cached. Do you think they will work on my problem? 2012/7/12 Aniket Mokashi <aniket...@gmail.com> > Can you share your query and use case? > > ~Aniket > > > On Tue, Jul 10, 2012 at 9:39 AM, Harsh J <ha...@cloudera.com> wrote: > >> This appears to be a Hive issue (something probably called FS.close() >> too early?). Redirecting to the Hive user lists as they can help >> better with this. >> >> On Tue, Jul 10, 2012 at 9:59 PM, 안의건 <ahneui...@gmail.com> wrote: >> > Hello. I have a problem with the filesystem closing. >> > >> > The filesystem was closed when the hive query is running. >> > It is 'select' query and the data size is about 1TB. >> > I'm using hadoop-0.20.2 and hive-0.7.1. >> > >> > The error log is telling that tmp file is not deleted, or the tmp path >> > exception is occurred. >> > >> > Is there any hadoop configuration I'm missing? >> > >> > Thank you >> > >> > [stderr logs] >> > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: >> > Filesystem closed >> > at >> > >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454) >> > at >> > >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566) >> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566) >> > at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193) >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) >> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) >> > at org.apache.hadoop.mapred.Child.main(Child.java:170) >> > Caused by: java.io.IOException: Filesystem closed >> > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226) >> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617) >> > at >> > >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453) >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648) >> > at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615) >> > at >> > >> org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68) >> > at >> > >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451) >> > ... 12 more >> >> >> >> -- >> Harsh J >> > > > > -- > "...:::Aniket:::... Quetzalco@tl" >