2008-09-25 17:12:18,250 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200809180916_0027_r_000007_2: Got 2 new map-outputs & number
of known map outputs is 21
2008-09-25 17:12:18,251 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200809180916_0027_r_000007_2 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2133)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.run(ReduceTask.java:2064)
Caused by: org.apache.lucene.index.MergePolicy$MergeException: segment
"_bfu exists in external directory yet the MergeScheduler executed the
merge in a separate thread
at
Hmm... could this be something specific to
org.apache.lucene.index.MergePolicy?
Arun
org
.apache
.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2362)
at
org
.apache
.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2307)
at
org
.apache
.hadoop
.contrib.index.mapred.IntermediateForm.process(IntermediateForm.java:
135)
at
org
.apache
.hadoop
.contrib
.index.mapred.IndexUpdateCombiner.reduce(IndexUpdateCombiner.java:56)
at
org
.apache
.hadoop
.contrib
.index.mapred.IndexUpdateCombiner.reduce(IndexUpdateCombiner.java:38)
at org.apache.hadoop.mapred.ReduceTask
$ReduceCopier.combineAndSpill(ReduceTask.java:2160)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
$3100(ReduceTask.java:341)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2120)
... 1 more
2008-09-25 17:12:19,087 INFO org.apache.hadoop.mapred.ReduceTask: Read
14523136 bytes from map-output for
attempt_200809180916_0027_m_000016_0
2008-09-25 17:12:19,087 INFO org.apache.hadoop.mapred.ReduceTask: Rec
#1 from attempt_200809180916_0027_m_000016_0 -> (41, 10651153) from
ars1dev6
2008-09-25 17:12:19,110 INFO org.apache.hadoop.mapred.ReduceTask:
Shuffling 14506735 bytes (14506735 raw bytes) into RAM from
attempt_200809180916_0027_m_000009_0
2008-09-25 17:12:19,226 INFO org.apache.hadoop.mapred.ReduceTask:
Closed ram manager
2008-09-25 17:12:19,228 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.io.IOException: attempt_200809180916_0027_r_000007_2The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:2209)
Thanks,
Joe
