I'll start with my support. I think it's fair to upgrade Spark versions to get new features at this point.
Questions: Since all the supported Spark versions are supported by a single artifact, do you drop the -3.0 in the artifact ID? Or leave it in case it's needed in the future? Does Flink still need Scala 2.11 support? I don't know much about Flink, but I guess that's self contained anyway so not a big deal either way? Adam On Sun, Jun 19, 2022, 2:10 AM Jia Yu <ji...@apache.org> wrote: > Dear all, > > I am proposing to drop the support of Spark 2.4 and Scala 2.11 in the next > Sedona release. The version number will be 1.3.0 if we drop this support, > otherwise it will be 1.2.1. > > Here is the status of Spark 2.4 and Sedona for Spark 2.4 > 1. Spark community has announced Spark 2.4 EOL on March 03 2021: > https://www.mail-archive.com/dev@spark.apache.org/msg27476.html > 2. Spark 3.0 was released on 06-16-2020. > 3. Spark 3.3.0 was released a few days ago. And starting from Spark 3.2, > Spark releases binaries for both Scala 2.12 and 2.13. > 4. Only a few Sedona users are using Spark 2.4. According to the statistics > of Maven Central (Scala/Java API only), only around 1K out of 100K > downloads are using Sedona for Spark 2.4. (core-2.4_2.11, core-2.4_2.12, > python-adapter-2.4_2.11, python-adapter-2.4_2.12) > > Benefits of dropping the support: > 1. Reduce the complexity of maintaining the source code for different Spark > versions. Currently, several files have two versions for Spark 2.4 and 3.x, > controlled by "anchor" keywords. I wrote a Python script to pre-process the > source code all the time: > > https://github.com/apache/incubator-sedona/blob/master/spark-version-converter.py > 2. Reduce the overhead of releasing binary packages. Currently, the main > POM.xml is quite complex in order to compile against different Spark > versions. Therefore, we weren't able to release Sedona for Scala 2.13. > > Plan of Sedona for Spark 3.X > 1. Sedona source code already supports Scala 2.13 but no Sedona binary > release. We will release Sedona for both Scala 2.12 and 2.13, but no Scala > 2.11. > 2. Sedona already releases binaries for Spark 3.0, 3.1, 3.2 > 3. The two latest PRs of Sedona are adding the support for Spark 3.3. > https://github.com/apache/incubator-sedona/pull/636 > https://github.com/apache/incubator-sedona/pull/635 > > What do you think of this proposal? If you don't like this, what is the > best time to drop the support of Spark 2.4 and Scala 2.11? > > I will let this discussion open for at least 3 days. If no objection, I > will remove Spark 2.4 from POM.xml and GitHub Actions, but leave the Spark > 2.4 support in the source code. So whoever wants to use Sedona on Spark 2.4 > can still compile the source code by themselves. > > Thanks, > Jia >
