Before OneHotEncoder or LabelIndexer is merged, you can define an UDF to do the mapping.
val labelToIndex = udf { ... } featureDF.withColumn("f3_dummy", labelToIndex(col("f3"))) See instructions here http://spark.apache.org/docs/latest/sql-programming-guide.html#udf-registration-moved-to-sqlcontextudf-java--scala -Xiangrui On Mon, Apr 6, 2015 at 7:31 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote: > Hi folks, currently have a DF that has a factor variable -- say gender. > > I am hoping to use the RandomForest algorithm on this data an it appears > that this needs to be converted to RDD[LabeledPoint] first -- i.e. all > features need to be double-encoded. > > I see https://issues.apache.org/jira/browse/SPARK-5888 is still open but was > wondering what is the recommended way to add a column? I can think of > > featuresDF.map { case Row(f1,f2,f3) =>(f1,f2,if (f3=='male') 0 else 1,if > (f3=='female') 0 else 1) }.toDF("f1","f2","f3_dummy","f3_dummy2") > > > but that isn't ideal as I already have 80+ features in that dataframe so the > matching itself is a pain -- thinking there's got to be a better way to > append |levels| number of columns and select all columns but "f3"? > > I see a withColumn method but no constructor to create a column...should I > be creating the dummy features in a new dataframe and then select them out > of there to get a Column? > > Any pointers are appreciated -- I'm sure I'm not the first person to attempt > this, just unsure of the least painful way to achieve. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org