Hi. you have a DataFrame...... there should be either a way to - convert a DF to a Vector without doing a cast - use a ML library which relies to DataFrames only
I can see that your code is still importing libraries from two different 'machine learning ' packages import org.apache.spark.ml.feature.{MinMaxScaler, Normalizer, StandardScaler, VectorAssembler} import org.apache.spark.mllib.linalg.{DenseVector, Vector, Vectors} You should be able to find exactly same data structures that you had in mllib under the ml package.....i'd advise to stick to ml libaries only, that will avoid confusion i concur with you, this line looks dodgy to me val rddVec = dfScaled .select("scaled_features") .rdd .map(_(0) .asInstanceOf[org.apache.spark.mllib.linalg.Vector]) converting a DF to a Vector is not as simple as doing a cast (like you would do in Java) I did a random search and found this, mayb it'll help https://community.hortonworks.com/questions/33375/how-to-convert-a-dataframe-to-a-vectordense-in-sca.html hth marco On Sat, Dec 31, 2016 at 4:24 AM, Jason Wolosonovich <jmwol...@asu.edu> wrote: > Hello All, > > I'm working through the Data Science with Scala course on Big Data > University and it is not updated to work with Spark 2.0, so I'm adapting > the code as I work through it, however I've finally run into something that > is over my head. I'm new to Scala as well. > > When I run this code (https://gist.github.com/jmwol > oso/a715cc4d7f1e7cc7951fab4edf6218b1) I get the following error: > > `java.lang.ClassCastException: org.apache.spark.ml.linalg.DenseVector > cannot be cast to org.apache.spark.mllib.linalg.Vector` > > I believe this is occurring at line 107 of the gist above. The code > starting at this line (and continuing to the end of the gist) is the > current code in the course. > > If I try to map to any other class type, then I have problems with the > `Statistics.corr(rddVec)`. > > How can I convert `rddVec` from an `ml.linalg.DenseVector` into an > `mllib.linalg.Vector` for use with `Statistics`? > > Thanks! > > -Jason > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >