DSv2 is far from stable right? All the actual data types are unstable and you guys have completely ignored that. We'd need to work on that and that will be a breaking change. If the goal is to make DSv2 work across 3.x and 2.x, that seems too invasive of a change to backport once you consider the parts needed to make dsv2 stable.
On Fri, Sep 20, 2019 at 10:47 AM, Ryan Blue < rb...@netflix.com.invalid > wrote: > > Hi everyone, > > > In the DSv2 sync this week, we talked about a possible Spark 2.5 release > based on the latest Spark 2.4, but with DSv2 and Java 11 support added. > > > A Spark 2.5 release with these two additions will help people migrate to > Spark 3.0 when it is released because they will be able to use a single > implementation for DSv2 sources that works in both 2.5 and 3.0. Similarly, > upgrading to 3.0 won't also require also updating to Java 11 because users > could update to Java 11 with the 2.5 release and have fewer major changes. > > > > Another reason to consider a 2.5 release is that many people are > interested in a release with the latest DSv2 API and support for DSv2 SQL. > I'm already going to be backporting DSv2 support to the Spark 2.4 line, so > it makes sense to share this work with the community. > > > This release line would just consist of backports like DSv2 and Java 11 > that assist compatibility, to keep the scope of the release small. The > purpose is to assist people moving to 3.0 and not distract from the 3.0 > release. > > > Would a Spark 2.5 release help anyone else? Are there any concerns about > this plan? > > > > > rb > > > > > -- > Ryan Blue > Software Engineer > Netflix >
