Hi,
I've been going over the kmeans stuff the last few days to try and
understand how it works, and how I might extend it to work with
the data I'm looking to process. It's taken me a while to get a
basic understanding of things, and really appreciate having lists
like this around for support.
I need to be able to label the vectors: each vector holds (for a
document) a set of similarity scores across a number of
attributes. I did some searching around payloads (after coming
across the term in some comments) but couldn't see how I add a
payload to the Vector. I then stumbled on MAHOUT-65 (https://issues.apache.org/jira/browse/MAHOUT-65
) that mentions the addition of the setName method to Vector. I've
tried building trunk, and although there were a few test failures
for other (seemingly unrelated) examples I continued and managed
to get the mahout-examples jar/job files built to give it a whirl.
When I run the following:
$ hadoop jar examples/target/mahout-examples-0.2-SNAPSHOT.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
I see it run the "Preparing Input", "Running Canopy to get initial
clusters", and then finally it starts "Running KMeans". But,
shortly after it breaks with the following trace:
---snip---
Running KMeans
09/07/13 23:49:34 INFO kmeans.KMeansDriver: Input: output/data
Clusters In: output/canopies Out: output Distance:
org.apache.mahout.utils.EuclideanDistanceMeasure
09/07/13 23:49:34 INFO kmeans.KMeansDriver: convergence: 0.5 max
Iterations: 10 num Reduce Tasks: 1 Input Vectors:
org.apache.mahout.matrix.SparseVector
09/07/13 23:49:34 INFO kmeans.KMeansDriver: Iteration 0
09/07/13 23:49:34 WARN mapred.JobClient: Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for
the same.
09/07/13 23:49:34 INFO mapred.FileInputFormat: Total input paths
to process : 2
09/07/13 23:49:34 INFO mapred.JobClient: Running job:
job_200907132019_0040
09/07/13 23:49:35 INFO mapred.JobClient: map 0% reduce 0%
09/07/13 23:49:42 INFO mapred.JobClient: map 50% reduce 0%
09/07/13 23:49:43 INFO mapred.JobClient: map 100% reduce 0%
09/07/13 23:49:49 INFO mapred.JobClient: map 100% reduce 100%
09/07/13 23:49:50 INFO mapred.JobClient: Job complete:
job_200907132019_0040
09/07/13 23:49:50 INFO mapred.JobClient: Counters: 16
09/07/13 23:49:50 INFO mapred.JobClient: File Systems
09/07/13 23:49:50 INFO mapred.JobClient: HDFS bytes read=465629
09/07/13 23:49:50 INFO mapred.JobClient: HDFS bytes written=5631
09/07/13 23:49:50 INFO mapred.JobClient: Local bytes read=7806
09/07/13 23:49:50 INFO mapred.JobClient: Local bytes
written=15674
09/07/13 23:49:50 INFO mapred.JobClient: Job Counters
09/07/13 23:49:50 INFO mapred.JobClient: Launched reduce tasks=1
09/07/13 23:49:50 INFO mapred.JobClient: Launched map tasks=2
09/07/13 23:49:50 INFO mapred.JobClient: Data-local map tasks=2
09/07/13 23:49:50 INFO mapred.JobClient: Map-Reduce Framework
09/07/13 23:49:50 INFO mapred.JobClient: Reduce input groups=7
09/07/13 23:49:50 INFO mapred.JobClient: Combine output
records=10
09/07/13 23:49:50 INFO mapred.JobClient: Map input records=600
09/07/13 23:49:50 INFO mapred.JobClient: Reduce output records=7
09/07/13 23:49:50 INFO mapred.JobClient: Map output bytes=465600
09/07/13 23:49:50 INFO mapred.JobClient: Map input bytes=448580
09/07/13 23:49:50 INFO mapred.JobClient: Combine input
records=600
09/07/13 23:49:50 INFO mapred.JobClient: Map output records=600
09/07/13 23:49:50 INFO mapred.JobClient: Reduce input records=10
09/07/13 23:49:50 WARN kmeans.KMeansDriver: java.io.IOException:
Cannot open filename /user/paul/output/clusters-0/_logs
java.io.IOException: Cannot open filename /user/paul/output/
clusters-0/_logs
at org.apache.hadoop.hdfs.DFSClient
$DFSInputStream.openInfo(DFSClient.java:1394)
at org.apache.hadoop.hdfs.DFSClient
$DFSInputStream.<init>(DFSClient.java:1385)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:338)
at
org
.apache
.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:
171)
at org.apache.hadoop.io.SequenceFile
$Reader.openFile(SequenceFile.java:1437)
at org.apache.hadoop.io.SequenceFile
$Reader.<init>(SequenceFile.java:1424)
at org.apache.hadoop.io.SequenceFile
$Reader.<init>(SequenceFile.java:1417)
at org.apache.hadoop.io.SequenceFile
$Reader.<init>(SequenceFile.java:1412)
at
org
.apache
.mahout
.clustering.kmeans.KMeansDriver.isConverged(KMeansDriver.java:304)
at
org
.apache
.mahout
.clustering.kmeans.KMeansDriver.runIteration(KMeansDriver.java:241)
at
org
.apache
.mahout.clustering.kmeans.KMeansDriver.runJob(KMeansDriver.java:194)
at
org
.apache
.mahout.clustering.syntheticcontrol.kmeans.Job.runJob(Job.java:100)
at
org
.apache
.mahout.clustering.syntheticcontrol.kmeans.Job.main(Job.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun
.reflect
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
---snip---
This is against revision 793689, running on my development Mac Pro
(pseudo-distributed single node) with Hadoop 0.19.1.
It's a bit late to be digging through what's going on, but will
try and take a look tomorrow- really excited about giving kmeans a
whirl on the document processing I'm playing with. In the
meantime, I was wondering whether anyone else had seen the same,
or knew a way to accomplish something similar with the released
version (or point me to a past good revision perhaps?)
Thanks again,
Paul
