Hello,
I wrote a MapReduce program which basically calculates Jaro Winkler distance
between each possible pair of strings of a string collection. The program
works fine for number of strings less than or equal to 8192. Then I tried
for 16384 strings but I am getting the following error which I have no clue
about:
09/02/24 14:05:22 INFO mapred.JobClient: map 100% reduce 0%
09/02/24 14:05:23 WARN mapred.LocalJobRunner: job_local_1
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for
taskTracker/jobcache/job_local_1/job_local_1_map_0000/output/file.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:982)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
Exception while running job: Job failed!
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
at
com.russoue.main.HadoopStringPairsMain.main(HadoopStringPairsMain.java:68)
I restarted the hadoop daemons several times and reran the program but the
problem remains. Can anyone tell me why this error might be happening?
Thanks,
--
Mohammad Farhan Husain
Research Assistant
Department of Computer Science
Erik Jonsson School of Engineering and Computer Science
University of Texas at Dallas