Eyal Allweil created DATAFU-168:
-----------------------------------
Summary: Add support for Spark 2.4.6 and up
Key: DATAFU-168
URL: https://issues.apache.org/jira/browse/DATAFU-168
Project: DataFu
Issue Type: Improvement
Affects Versions: 1.6.1
Reporter: Eyal Allweil
Fix For: 1.7.0
Once DATAFU-167 is merged, datafu-spark will support Spark versions up to
2.4.5. However, because our implementation of _collectLimitedList_ extends
Spark's {_}collect{_}, and because its interface was changed in 2.4.6,
compilation is broken for us.
(here is the relevant line from collectLimitedList:
[https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkOverwriteUDAFs.scala#L104)]
We need to either *1)* update our implementation, and drop support for older
versions (and then release this in our version 1.8.0) or *2)* copy the code in
a backwards compatible way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
