You will get 10x speedup by not using mahout vector and use breeze sparse vector from mllib in your mllib kmeans run....
@Xiangrui showed the comparison chart sometime back... On May 14, 2014 6:33 AM, "Xiangrui Meng" <men...@gmail.com> wrote: > You need > > > val raw = sc.sequenceFile(path, classOf[Text], classOf[VectorWriteable]) > > to load the data. After that, you can do > > > val data = raw.values.map(_.get) > > To get an RDD of mahout's Vector. You can use `--jar mahout-math.jar` > when you launch spark-shell to include mahout-math. > > Best, > Xiangrui > > On Tue, May 13, 2014 at 10:37 PM, Stuti Awasthi <stutiawas...@hcl.com> > wrote: > > Hi All, > > > > I am very new to Spark and trying to play around with Mllib hence > apologies > > for the basic question. > > > > > > > > I am trying to run KMeans algorithm using Mahout and Spark MLlib to see > the > > performance. Now initial datasize was 10 GB. Mahout converts the data in > > Sequence File <Text,VectorWritable> which is used for KMeans Clustering. > > The Sequence File crated was ~ 6GB in size. > > > > > > > > Now I wanted if I can use the Mahout Sequence file to be executed in > Spark > > MLlib for KMeans . I have read that SparkContext.sequenceFile may be used > > here. Hence I tried to read my sequencefile as below but getting the > error : > > > > > > > > Command on Spark Shell : > > > > scala> val data = sc.sequenceFile[String,VectorWritable]("/ > > KMeans_dataset_seq/part-r-00000",String,VectorWritable) > > > > <console>:12: error: not found: type VectorWritable > > > > val data = sc.sequenceFile[String,VectorWritable](" > > /KMeans_dataset_seq/part-r-00000",String,VectorWritable) > > > > > > > > Here I have 2 ques: > > > > 1. Mahout has “Text” as Key but Spark is printing “not found: type:Text” > > hence I changed it to String.. Is this correct ??? > > > > 2. How will VectorWritable be found in Spark. Do I need to include Mahout > > jar in Classpath or any other option ?? > > > > > > > > Please Suggest > > > > > > > > Regards > > > > Stuti Awasthi > > > > > > > > ::DISCLAIMER:: > > > ---------------------------------------------------------------------------------------------------------------------------------------------------- > > > > The contents of this e-mail and any attachment(s) are confidential and > > intended for the named recipient(s) only. > > E-mail transmission is not guaranteed to be secure or error-free as > > information could be intercepted, corrupted, > > lost, destroyed, arrive late or incomplete, or may contain viruses in > > transmission. The e mail and its contents > > (with or without referred errors) shall therefore not attach any > liability > > on the originator or HCL or its affiliates. > > Views or opinions, if any, presented in this email are solely those of > the > > author and may not necessarily reflect the > > views or opinions of HCL or its affiliates. Any form of reproduction, > > dissemination, copying, disclosure, modification, > > distribution and / or publication of this message without the prior > written > > consent of authorized representative of > > HCL is strictly prohibited. If you have received this email in error > please > > delete it and notify the sender immediately. > > Before opening any email and/or attachments, please check them for > viruses > > and other defects. > > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------- >