Hi, I also think that we need to support 2.4.
I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep 12, 2020) I also asked about it on [email protected] : Sean Owen (answered the question): "I don't think there's an official EOL for Spark 2.4.x but would expect another maintenance release in the first half of 2021 at least. I'd also guess it wouldn't be maintained by 2022." ?BR, Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:[email protected]] ________________________________ From: Paweł Kociński <[email protected]> Sent: Wednesday, November 11, 2020 00:29 To: Jia Yu Cc: [email protected]; [email protected]; Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Zongsi Zhang Subject: Re: Support Spark 2.4 in Sedona 1.0 Hi Jia, I think we should support spark 2.4, a lot of users still use it. More than that I think more users still have jobs written in spark 2.4 than 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 with scala 2.12 is important one. Regards, Paweł pon., 9 lis 2020 o 20:44 Jia Yu <[email protected]<mailto:[email protected]>> napisał(a): Dear all, In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether we should support Spark 2.4. In order to support Spark 2.4, we need to do the following 1. Compile the source using Scala 2.11. Sedona master branch currently is compiled by Scala 2.12 and Java 1.8 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) UDF registration hook (2) the SQL aggregation function format 3. In the future releases of Sedona, use git cherry-pick to pick important features back to the Spark 2.4 branch. This is what I did in GeoSpark to support Spark 2.1, 2.2, 2.3 GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it that way and just support Spark 3.0. Do you think we should support Spark 2.4 in the future release? Thanks, Jia Yu
