I use Spark 1.5 with CDH5.5 distribution and I see that support is present for UDAF. From the link: https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html, I read that this is an experimental feature. So it makes sense not to find this in the documentation.
For confirmation whether it works in Spark 1.5 I quickly tried out the example in the link and it works. I hope this answers your question. Kabeer. On 18/02/16 16:31, Richard Cobbe wrote: I'm working on an application using DataFrames (Scala API) in Spark 1.5.0, and we need to define and use several custom aggregators. I'm having trouble figuring out how to do this, however. First, which version of Spark did UDAF support land in? Has it in fact landed at all? https://issues.apache.org/jira/browse/SPARK-3947 suggests that UDAFs should be available in 1.5.0. However, the associated pull request includes classes like org.apache.spark.sql.UDAFRegistration, but these classes don't appear in the API docs, and I'm not able to use them from the spark shell ("type UDAFRegistration is not a member of package org.apache.spark.sql"). I don't have access to a Spark 1.6.0 installation, but UDAFRegistration doesn't appear in the Scaladoc pages for 1.6. Second, assuming that this functionality is supported in some version of Spark, could someone point me to some documentation or an example that demonstrates how to define and use a custom aggregation function? Many thanks, Richard --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
