Re: FileSystem Closed.

2012-07-26 Thread 안의건 
Thank you

I've been researching based on your opinions, and found the below two
solutions.
These are the answers for who has FileSystem.closed issue like me.


 -  close it in your cleanup method and you have JVM reuse turned on
(mapred.job.reuse.jvm.num.tasks)

 - set "fs.hdfs.impl.disable,cache' to turn in the conf, and new instances
don't get cached.


Do you think they will work on my problem?

2012/7/12 Aniket Mokashi 

> Can you share your query and use case?
>
> ~Aniket
>
>
> On Tue, Jul 10, 2012 at 9:39 AM, Harsh J  wrote:
>
>> This appears to be a Hive issue (something probably called FS.close()
>> too early?). Redirecting to the Hive user lists as they can help
>> better with this.
>>
>> On Tue, Jul 10, 2012 at 9:59 PM, 안의건  wrote:
>> > Hello. I have a problem with the filesystem closing.
>> >
>> > The filesystem was closed when the hive query is running.
>> > It is 'select' query and the data size is about 1TB.
>> > I'm using hadoop-0.20.2 and hive-0.7.1.
>> >
>> > The error log is telling that tmp file is not deleted, or the tmp path
>> > exception is occurred.
>> >
>> > Is there any hadoop configuration I'm missing?
>> >
>> > Thank you
>> >
>> > [stderr logs]
>> > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
>> > Filesystem closed
>> > at
>> >
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454)
>> > at
>> >
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
>> > at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
>> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> > Caused by: java.io.IOException: Filesystem closed
>> > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
>> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
>> > at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
>> > at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615)
>> > at
>> >
>> org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68)
>> > at
>> >
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451)
>> > ... 12 more
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>

Re: FileSystem Closed.

2012-07-10 Thread Harsh J 
This appears to be a Hive issue (something probably called FS.close()
too early?). Redirecting to the Hive user lists as they can help
better with this.

On Tue, Jul 10, 2012 at 9:59 PM, 안의건  wrote:
> Hello. I have a problem with the filesystem closing.
>
> The filesystem was closed when the hive query is running.
> It is 'select' query and the data size is about 1TB.
> I'm using hadoop-0.20.2 and hive-0.7.1.
>
> The error log is telling that tmp file is not deleted, or the tmp path
> exception is occurred.
>
> Is there any hadoop configuration I'm missing?
>
> Thank you
>
> [stderr logs]
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> Filesystem closed
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
> at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615)
> at
> org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451)
> ... 12 more



-- 
Harsh J

Re: FileSystem closed

2011-09-30 Thread Steve Loughran 

On 29/09/2011 18:02, Joey Echeverria wrote:

Do you close your FileSystem instances at all? IIRC, the FileSystem
instance you use is a singleton and if you close it once, it's closed
for everybody. My guess is you close it in your cleanup method and you
have JVM reuse turned on.



I've hit this in the past. In 0.21+ you can ask for a new instance 
explicity.


For 0.20.20x, set "fs.hdfs.impl.disable.cache" to true in the conf, and 
new instances don't get cached.

Re: FileSystem closed

2011-09-29 Thread Uma Maheswara Rao G 72686 
FileSystem objects will be cached in jvm.
When it tries to get the FS object by using Filesystem.get(..) ( sequence file 
internally will use it), it will return same fs object if scheme and authority 
is same for the uri.

 fs cache key's equals implementation is below

 static boolean isEqual(Object a, Object b) {
return a == b || (a != null && a.equals(b));
  }

  /** {@inheritDoc} */
  public boolean equals(Object obj) {
if (obj == this) {
  return true;
}
if (obj != null && obj instanceof Key) {
  Key that = (Key)obj;
  return isEqual(this.scheme, that.scheme)
 && isEqual(this.authority, that.authority)
 && isEqual(this.ugi, that.ugi)
 && (this.unique == that.unique);
}
return false;
  }


I think, here some your files uri and schems are same and got the same fs 
object. When it closes first one, diffenitely other will get this exception.

Regards,
Uma

- Original Message -
From: Joey Echeverria 
Date: Thursday, September 29, 2011 10:34 pm
Subject: Re: FileSystem closed
To: common-user@hadoop.apache.org

> Do you close your FileSystem instances at all? IIRC, the FileSystem
> instance you use is a singleton and if you close it once, it's closed
> for everybody. My guess is you close it in your cleanup method and you
> have JVM reuse turned on.
> 
> -Joey
> 
> On Thu, Sep 29, 2011 at 12:49 PM, Mark question 
>  wrote:
> > Hello,
> >
> >  I'm running 100 mappers sequentially on a single machine, where 
> each> mapper opens 100 files at the beginning then read one by one 
> sequentially> and closes after each one is done. After executing 6 
> mappers, the 7th gives
> > this error:
> >
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at 
> java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at 
> Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> > 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>
>     at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at 
> java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at 
> Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> > 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>
>     at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at 
> java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at 
> Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.ru

Re: FileSystem closed

2011-09-29 Thread Joey Echeverria 
Do you close your FileSystem instances at all? IIRC, the FileSystem
instance you use is a singleton and if you close it once, it's closed
for everybody. My guess is you close it in your cleanup method and you
have JVM reuse turned on.

-Joey

On Thu, Sep 29, 2011 at 12:49 PM, Mark question  wrote:
> Hello,
>
>  I'm running 100 mappers sequentially on a single machine, where each
> mapper opens 100 files at the beginning then read one by one sequentially
> and closes after each one is done. After executing 6 mappers, the 7th gives
> this error:
>
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>
>
> Can anybody give me a hint of what that could be?
>
> Thank you,
> Mark
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434