Forking this thread to muse about Spark 3. Like Spark 2, I assume it would be more about making all those accumulated breaking changes and updating lots of dependencies. Hadoop 3 looms large in that list as well as Scala 2.12.
Spark 1 was release in May 2014, and Spark 2 in July 2016. If Spark 2.3 is out in Feb 2018 and it takes the now-usual 6 months until a next release, Spark 3 could reasonably be next. However the release cycles are naturally slowing down, and it could also be said that 2019 would be more on schedule for Spark 3. Nothing particularly urgent about deciding, but I'm curious if anyone had an opinion on whether to move on to Spark 3 next or just continue with 2.4 later this year. On Fri, Jan 19, 2018 at 11:13 AM Sean Owen <so...@cloudera.com> wrote: > Yeah, if users are using Kryo directly, they should be insulated from a > Spark-side change because of shading. > However this also entails updating (unshaded) Chill from 0.8.x to 0.9.x. I > am not sure if that causes problems for apps. > > Normally I'd avoid any major-version change in a minor release. This one > looked potentially entirely internal. > I think if there are any doubts, we can leave it for Spark 3. There was a > bug report that needed a fix from Kryo 4, but it might be minor after all. > >> >>
