Dear all,
I use Hadoop 0.18.0 to execute a job which will output huge key-value pairs
in Mapper phase. While running the job in the mapper phase, the hadoop
framework throws exception as below:
java.io.FileNotFoundException:
/home/guangfeng/bin/hadoop-0.18.0/tmp/hadoop-guangfeng/mapred/local/taskTracker/jobcache/job_200810211354_0001/attempt_200810211354_0001_m_000005_0/output/spill491.out.index
(Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>(RawLocalFileSystem.java:70)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:106)
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:176)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:117)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:274)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:375)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1025)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:702)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
It seems that the framework opens too many files at the same time which
exceeds the linux file system limits. Anybody faced with such problems
before? By the way, can anybody give a detailed process description while
running mappers?
Thanks
--
Guangfeng Jin
Software Engineer
