[
https://issues.apache.org/jira/browse/DATAFU-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771358#comment-17771358
]
Eyal Allweil commented on DATAFU-169:
-------------------------------------
I opened [a pull request|https://github.com/apache/datafu/pull/38] for which
tests pass for Spark 3.0.x and 3.1.x.
It seems like our use of _UserDefinedAggregateFunction_ is preventing us from
proceeding to Spark 3.2.x - I've opened [this
issue|https://issues.apache.org/jira/browse/DATAFU-173] for upgrading to
{_}Aggregator{_}.
> Support Spark 3.x
> -----------------
>
> Key: DATAFU-169
> URL: https://issues.apache.org/jira/browse/DATAFU-169
> Project: DataFu
> Issue Type: Improvement
> Reporter: Eyal Allweil
> Assignee: Eyal Allweil
> Priority: Major
> Fix For: 2.0.0
>
>
> This is our umbrella-JIRA-issue for covering the changes we'll need to make
> to support Spark 3.
>
> There are some linked issues for tasks that need to be done to get this to
> work. It doesn't look like there are that many actual code changes to our
> actual production code.
>
> To begin with, let's aim to support Spark 3.0.x and 3.1.x and work our way to
> newer versions.
>
> Here is the guide to doing this upgrade:
> [https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
