Before OneHotEncoder or LabelIndexer is merged, you can define an UDF
to do the mapping.
val labelToIndex = udf { ... }
featureDF.withColumn("f3_dummy", labelToIndex(col("f3")))
See instructions here
http://spark.apache.org/docs/latest/sql-programming-guide.html#udf-registration-moved-to-sqlcontextudf-java--scala
-Xiangrui
On Mon, Apr 6, 2015 at 7:31 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote:
> Hi folks, currently have a DF that has a factor variable -- say gender.
>
> I am hoping to use the RandomForest algorithm on this data an it appears
> that this needs to be converted to RDD[LabeledPoint] first -- i.e. all
> features need to be double-encoded.
>
> I see https://issues.apache.org/jira/browse/SPARK-5888 is still open but was
> wondering what is the recommended way to add a column? I can think of
>
> featuresDF.map { case Row(f1,f2,f3) =>(f1,f2,if (f3=='male') 0 else 1,if
> (f3=='female') 0 else 1) }.toDF("f1","f2","f3_dummy","f3_dummy2")
>
>
> but that isn't ideal as I already have 80+ features in that dataframe so the
> matching itself is a pain -- thinking there's got to be a better way to
> append |levels| number of columns and select all columns but "f3"?
>
> I see a withColumn method but no constructor to create a column...should I
> be creating the dummy features in a new dataframe and then select them out
> of there to get a Column?
>
> Any pointers are appreciated -- I'm sure I'm not the first person to attempt
> this, just unsure of the least painful way to achieve.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
