Hi everyone, In the DSv2 sync this week, we talked about a possible Spark 2.5 release based on the latest Spark 2.4, but with DSv2 and Java 11 support added.
A Spark 2.5 release with these two additions will help people migrate to Spark 3.0 when it is released because they will be able to use a single implementation for DSv2 sources that works in both 2.5 and 3.0. Similarly, upgrading to 3.0 won't also require also updating to Java 11 because users could update to Java 11 with the 2.5 release and have fewer major changes. Another reason to consider a 2.5 release is that many people are interested in a release with the latest DSv2 API and support for DSv2 SQL. I'm already going to be backporting DSv2 support to the Spark 2.4 line, so it makes sense to share this work with the community. This release line would just consist of backports like DSv2 and Java 11 that assist compatibility, to keep the scope of the release small. The purpose is to assist people moving to 3.0 and not distract from the 3.0 release. Would a Spark 2.5 release help anyone else? Are there any concerns about this plan? rb -- Ryan Blue Software Engineer Netflix
