I'm a mahout novice trying to do some semantic data clustering with
Canopy clustering on some low-dimensional SequenceFiles that I
vectorized with ad-hoc java code. (Some features are strings
vextorized by the Levenstein distance from a constant, some are
DateTime objects vectorized as milliseconds from the Unix era, some
are georeferences, etc. etc.). The results look promising, but I want
to get more detail out of the clusters, than I understand how to get
from ClusterDumper alone. In particulary, it seems that
CSVClusterWriter should get me what I need (for each cluster, the
center and the list of vectors ordered by distance.
When I vectorized, I never explicitly built a Dictionary, which is---I
suppose---why I get a runtime ClassCastException when I invoke
ClusterDumper.readPoints(...), despite telling the ClusterDumper run
method that the dictionary type is "sequencefile", but have no
sequencefile to offer.
So I have these questions:
1. Am I right that the Exception in the dumper is caused by not having
a Dictionary file?
2. Where can I find documentation for the correct form of a
sequencefile Dictionary and are there any convenience methods for
building it? I start with a CSV file for the data, together with a Map
that associates column header names with a private type name that
specifies the algorithm to be applied to the vectorization.) I can
send the vectorization if helpful.
Thanks in advance;
--Bob
Here's the dumper code with point of ClassCastException indicated
public void test() throws Exception {
String datasetDir = "Lichen/"; // bbg, Rubiaeceae/ or fungi/ for now
String inputFile = "/tmp/vectors"; //inputDir + "vectors"; // input,
String canopyOutput = "/tmp/clusters";
String dumperInput = canopyOutput+"/clusters-0-final";
String dumperOutput = "/tmp/clusters.txt";
String clusterInput = dumperInput+"/"+"part-r-00000";
String clusterOutput = "/tmp/clusterDetail.txt";
boolean runSequential = true;
try {
String[] args = {"-i", inputFile, "-o", canopyOutput, "-t1",
".00000002", "-t2", ".00000001", "-ow"};
CanopyDriver driver = new CanopyDriver();
driver.run(args);
//must need Path to the sequence file here also?
String[] dumpArgs = {"-i", dumperInput, "-o", dumperOutput,
"-dt", "sequencefile"};
ClusterDumper dumper = new ClusterDumper();
dumper.run(dumpArgs);
PrintWriter writer = new PrintWriter(new File(clusterOutput));
Path pointsPathDir = new Path(dumperInput);
Configuration conf=new Configuration();
////// Line below throws runtime
////
////// java.lang.ClassCastException:
org.apache.hadoop.io.Text cannot be cast to
org.apache.hadoop.io.IntWritable////
// Presumably need a Dictionary to pass via -d to ClusterDumper
Map<Integer, List<WeightedPropertyVectorWritable>> clusterIdToPoints =
ClusterDumper.readPoints(pointsPathDir, 10000, conf);
//TODO: iterate over Map and output with CSVClusterWriter
csvClusterWriter = new CSVClusterWriter(writer, clusterIdToPoints,
measure);
} catch (Exception e) {
System.out.println("test caught Exception");
e.printStackTrace(System.out);
}
}
--
Robert A. Morris
Emeritus Professor of Computer Science
UMASS-Boston
100 Morrissey Blvd
Boston, MA 02125-3390
Filtered Push Project
Harvard University Herbaria
Harvard University
email: morris....@gmail.com
web: http://efg.cs.umb.edu/
web: http://wiki.filteredpush.org
http://www.cs.umb.edu/~ram
===
The content of this communication is made entirely on my
own behalf and in no way should be deemed to express
official positions of The University of Massachusetts at Boston or
Harvard University.
