[
https://issues.apache.org/jira/browse/MAHOUT-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674440#comment-13674440
]
yannis ats commented on MAHOUT-1233:
------------------------------------
Hi
i am doing clustering with canopy and i get correct results when i do the
following:
$mahout_dir canopy -i $input_dir -o $temp_dir -dm
org.apache.mahout.common.distance.CosineDistanceMeasure -t1 $T1 -t2 $T2 -cl -xm
mapreduce -ow --tempDir $temp_dir/temp
where input_dir can be one single chunk .mvc file or a bunch of them.
now after the clustering i do the following:
partial_command="${localtemp_dir}/`hadoop fs -ls $localtemp_dir | grep -o
"clusters-.*-final"`"
echo $partial_command
$mahout_dir seqdumper -i $partial_command -o
$LOCALTEMP_DIR/clustered_points1.txt --tempDir $temp_dir/temp
$mahout_dir clusterdump -i $partial_command -o
$LOCALTEMP_DIR/clustered_points2.txt -dt sequencefile -n 20 -dm
org.apache.mahout.common.distance.CosineDistanceMeasure -sp 0 --pointsDir
$localtemp_dir/clusteredPoints --tempDir $temp_dir/temp
tail -5 $LOCALTEMP_DIR/clustered_points2.txt >> "$results_file"
cluster_center_coordinates="$results_dir/cluster_center_coordinates.txt"
awk '/C-/' $LOCALTEMP_DIR/clustered_points2.txt | grep -P -o
"(C-[0-9]+)|(\[.+\] )" >> "$cluster_center_coordinates"
#Get the clustered points and store them
clustered_points="$results_dir/clustered_points.txt"
partial_list1=`hadoop fs -ls $localtemp_dir/clusteredPoints | grep -o "part-.*"`
for variable in $partial_list1
do
variable1="${localtemp_dir}/clusteredPoints/$variable"
echo $variable1
$mahout_dir seqdumper -i $variable1 -o $LOCALTEMP_DIR/temp_clustered.txt
--tempDir $temp_dir/temp
#cat $LOCALTEMP_DIR/temp_clustered.txt >>$LOCALTEMP_DIR/clustered_points2.txt
awk 'NR==3,EOF' $LOCALTEMP_DIR/temp_clustered.txt | sed '$d' >
$LOCALTEMP_DIR/points_temp1.txt
#
cat $LOCALTEMP_DIR/points_temp1.txt >>$LOCALTEMP_DIR/points.txt
in which i create a file points.txt where i have all the information needed to
get the clusterid,vectorid
i have a program in python for this processing and is correct.
SO to give you an overview the aforementioned code is working when the data is
one chunk or multiple chunks
for example if i have 10000 vectors to cluster with canopy with the
aforementioned code if is 10000*1 or 1000*10 chunks
then i get a file points.txt with 10000 records
Now the problem starts when i have the meanshift algorithm(and i think in
kmeans and fuzzy kmeans)
in particular i did the same for meanshift
and the final file points.txt has more than 10000 records(vectors)
The thing is that i was obtaining same errors for canopy when instead of using
as input the part-m-00001,part-m-00000 file i was giving the name of the
directory that they exist.
i thought that the trick of using seqdumper per chunk would solve my problems
and it was successful(the code that i just provided is successful) for canopy
but when i go to meanshift i get more point ids..
do you think that i have done something fundamentally wrong?
ALso i know that my data has well clustered form but when i did the meanshift
with T1=0.1 and T2=0.1 i get more clusters than the #of points.
any help is appreciated.
kind regards
yannis
> Problem in processing datasets as a single chunk vs many chunks in HADOOP
> mode in mostly all the clustering algos
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1233
> URL: https://issues.apache.org/jira/browse/MAHOUT-1233
> Project: Mahout
> Issue Type: Question
> Components: Clustering
> Affects Versions: 0.7, 0.8
> Reporter: yannis ats
> Assignee: yannis ats
> Priority: Minor
> Fix For: 0.8
>
>
> I am trying to process a dataset and i do it in two ways.
> Firstly i give it as a single chunk(all the dataset) and secondly as many
> smaller chunks in order to increase the throughput of my machine.
> The problem is that when i perform the single chunk computation the results
> are fine
> and by fine i mean that if i have in the input 1000 vectors i get in the
> output 1000 vectorids with their cluster_ids (i have tried in canopy,kmeans
> and fuzzy kmeans).
> However when i split the dataset in order to speed up the computations then
> strange phenomena occur.
> For instance the same dataset that contains 1000 vectors and is split in for
> example 10 files then in the output i will obtain more vector ids(w.g 1100
> vectorids with their corresponding clusterids).
> The question is, am i doing something wrong in the process?
> Is there a problem in clusterdump and seqdumper when the input is in many
> files?
> I have observed when mahout is performing the computations that in the screen
> says that processed the correct number of vectors.
> Am i missing something?
> I use as input the transformed to mvc weka vectors.
> I have tried this in v0.7 and the v0.8 snapshot.
> Thank you in advance for your time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
