[
https://issues.apache.org/jira/browse/SPARK-30520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136153#comment-17136153
]
Erik Erlandson commented on SPARK-30520:
----------------------------------------
Starting in spark 3.0, any custom aggregator that would have been implemented
using UserDefinedAggregateFunction should now be implemented using Aggregator.
To use a custom Aggregator with a dynamically typed DataFrame (aka
DataSet[Row]), register it using org.apache.spark.sql.functions.udaf
> Eliminate deprecation warnings for UserDefinedAggregateFunction
> ---------------------------------------------------------------
>
> Key: SPARK-30520
> URL: https://issues.apache.org/jira/browse/SPARK-30520
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Major
>
> {code}
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala
> Warning:Warning:line (718)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> val udaf =
> clazz.getConstructor().newInstance().asInstanceOf[UserDefinedAggregateFunction]
> Warning:Warning:line (719)method register in class UDFRegistration is
> deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now be registered
> as a UDF via the functions.udaf(agg) method.
> register(name, udaf)
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala
> Warning:Warning:line (328)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> udaf: UserDefinedAggregateFunction,
> Warning:Warning:line (326)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> case class ScalaUDAF(
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala
> Warning:Warning:line (363)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> val udaf = new UserDefinedAggregateFunction {
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleSum.java
> Warning:Warning:line (25)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> Warning:Warning:line (35)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/core/src/test/java/test/org/apache/spark/sql/MyDoubleAvg.java
> Warning:Warning:line (25)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> Warning:Warning:line (36)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
> Warning:Warning:line (36)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> class ScalaAggregateFunction(schema: StructType) extends
> UserDefinedAggregateFunction {
> Warning:Warning:line (73)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> class ScalaAggregateFunctionWithoutInputSchema extends
> UserDefinedAggregateFunction {
> Warning:Warning:line (100)class UserDefinedAggregateFunction in package
> expressions is deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now
> be registered as a UDF via the functions.udaf(agg) method.
> class LongProductSum extends UserDefinedAggregateFunction {
> Warning:Warning:line (189)method register in class UDFRegistration is
> deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now be registered
> as a UDF via the functions.udaf(agg) method.
> spark.udf.register("mydoublesum", new MyDoubleSum)
> Warning:Warning:line (190)method register in class UDFRegistration is
> deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now be registered
> as a UDF via the functions.udaf(agg) method.
> spark.udf.register("mydoubleavg", new MyDoubleAvg)
> Warning:Warning:line (191)method register in class UDFRegistration is
> deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now be registered
> as a UDF via the functions.udaf(agg) method.
> spark.udf.register("longProductSum", new LongProductSum)
> Warning:Warning:line (943)method register in class UDFRegistration is
> deprecated (since 3.0.0): Aggregator[IN, BUF, OUT] should now be registered
> as a UDF via the functions.udaf(agg) method.
> spark.udf.register("noInputSchema", new
> ScalaAggregateFunctionWithoutInputSchema)
> /Users/maxim/proj/eliminate-expr-info-warnings/sql/hive/src/test/java/org/apache/spark/sql/hive/JavaDataFrameSuite.java
> Warning:Warning:line (30)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> Warning:Warning:line (78)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> Warning:Warning:line (79)java:
> org.apache.spark.sql.expressions.UserDefinedAggregateFunction in
> org.apache.spark.sql.expressions has been deprecated
> Warning:Warning:line (79)java:
> register(java.lang.String,org.apache.spark.sql.expressions.UserDefinedAggregateFunction)
> in org.apache.spark.sql.UDFRegistration has been deprecated
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
