[
https://issues.apache.org/jira/browse/MAHOUT-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899416#action_12899416
]
Han Hui Wen commented on MAHOUT-471:
-------------------------------------
{code}
public class DistributedTanimotoCoefficientVectorSimilarity extends
AbstractDistributedVectorSimilarity {
@Override
protected double doComputeResult(int rowA, int rowB, Iterable<Cooccurrence>
cooccurrences, double weightOfVectorA,
double weightOfVectorB, int numberOfColumns) {
double cooccurrenceCount = countElements(cooccurrences);
if (cooccurrenceCount == 0) {
return Double.NaN;
}
return cooccurrenceCount / (weightOfVectorA + weightOfVectorB -
cooccurrenceCount);
}
@Override
public double weight(Vector v) {
return (double) countElements(v.iterateNonZero());
}
}
{code}
{code}
public abstract class AbstractDistributedVectorSimilarity implements
DistributedVectorSimilarity {
/**
* ensures that the computed similarity is in [-1,1]
*/
@Override
public final double similarity(int rowA, int rowB, Iterable<Cooccurrence>
cooccurrences, double weightOfVectorA,
double weightOfVectorB, int numberOfColumns) {
double result = doComputeResult(rowA, rowB, cooccurrences, weightOfVectorA,
weightOfVectorB, numberOfColumns);
if (result < -1.0) {
result = -1.0;
} else if (result > 1.0) {
result = 1.0;
}
return result;
}
{code}
If doComputeResult == Double.NaN, then similarity will return Double.NaN.
Maybe Double.NaN. cause the issue.
> RowSimilarityJob-Mapper-EntriesToVectorsReducer failure
> --------------------------------------------------------
>
> Key: MAHOUT-471
> URL: https://issues.apache.org/jira/browse/MAHOUT-471
> Project: Mahout
> Issue Type: Bug
> Components: Collaborative Filtering
> Affects Versions: 0.4
> Reporter: Han Hui Wen
> Priority: Minor
> Fix For: 0.4
>
>
> I used Boolean Data and SIMILARITY_TANIMOTO_COEFFICIENT
> java.io.IOException: Task: attempt_201008101359_0084_r_000000_0 - The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.io.IOException: Intermediate merge failed
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2576)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2501)
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:103)
> at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
> at org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:123)
> at org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:50)
> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:447)
> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:381)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:107)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:93)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2551)
> ... 1 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.mahout.math.Varint.readUnsignedVarInt(Varint.java:159)
> at org.apache.mahout.math.Varint.readSignedVarInt(Varint.java:140)
> at
> org.apache.mahout.math.hadoop.similarity.SimilarityMatrixEntryKey.readFields(SimilarityMatrixEntryKey.java:65)
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:97)
> ... 9 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
