Hi,
Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with some
of our clustering code and Hadoop 0.18.1. The thread in context is
at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
The problem seems to occur when going from 0.17.2 to 0.18.1. In the
user logs, we are seeing the following exception:
2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 5011 bytes
2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810272112_0011_r_000000_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.run(ReduceTask.java:2078)
Caused by: java.lang.NumberFormatException: For input string: "["
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
at java.lang.Double.parseDouble(Double.java:510)
at
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
at
org
.apache.mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:
256)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
at org.apache.hadoop.mapred.ReduceTask
$ReduceCopier.combineAndSpill(ReduceTask.java:2174)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
$3100(ReduceTask.java:341)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
And in the main output log (from running bin/hadoop jar mahout/
examples/build/apache-mahout-examples-0.1-dev.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
attempt_200810272112_0011_r_000000_0, Status : FAILED
java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:2207)
If I run this exact same job on 0.17.2 it all runs fine. I suppose
either a bug was introduced in 0.18.1 or a bug was fixed that we were
relying on. Looking at the release notes between the fixes, nothing
in particular struck me as related. If it helps, I can provide the
instructions for how to run the example in question (they need to be
written up anyway!)
I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
, but those are older, it seems, so not sure what to make of them.
Thanks,
Grant
