+1 to Scala 2.12 as the default in Spark 3.0. On Tue, Nov 6, 2018 at 11:50 AM DB Tsai <d_t...@apple.com> wrote:
> +1 on dropping Scala 2.11 in Spark 3.0 to simplify the build. > > As Scala 2.11 will not support Java 11 unless we make a significant > investment, if we decide not to drop Scala 2.11 in Spark 3.0, what we can > do is have only Scala 2.12 build support Java 11 while Scala 2.11 support > Java 8. But I agree with Sean that this can make the decencies really > complicated; hence I support to drop Scala 2.11 in Spark 3.0 directly. > > DB Tsai | Siri Open Source Technologies [not a contribution] | > Apple, Inc > > On Nov 6, 2018, at 11:38 AM, Sean Owen <sro...@gmail.com> wrote: > > I think we should make Scala 2.12 the default in Spark 3.0. I would > also prefer to drop Scala 2.11 support in 3.0. In theory, not dropping > 2.11 support it means we'd support Scala 2.11 for years, the lifetime > of Spark 3.x. In practice, we could drop 2.11 support in a 3.1.0 or > 3.2.0 release, kind of like what happened with 2.10 in 2.x. > > Java (9-)11 support also complicates this. I think getting it to work > will need some significant dependency updates, and I worry not all > will be available for 2.11 or will present some knotty problems. We'll > find out soon if that forces the issue. > > Also note that Scala 2.13 is pretty close to release, and we'll want > to support it soon after release, perhaps sooner than the long delay > before 2.12 was supported (because it was hard!). It will probably be > out well before Spark 3.0. Cross-compiling for 3 Scala versions sounds > like too much. 3.0 could support 2.11 and 2.12, and 3.1 support 2.12 > and 2.13, or something. But if 2.13 support is otherwise attainable at > the release of Spark 3.0, I wonder if that too argues for dropping > 2.11 support. > > Finally I'll say that Spark itself isn't dropping 2.11 support for a > while, no matter what; it still exists in the 2.4.x branch of course. > People who can't update off Scala 2.11 can stay on Spark 2.x, note. > > Sean > > > On Tue, Nov 6, 2018 at 1:13 PM DB Tsai <d_t...@apple.com> wrote: > > > We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next > Spark version will be 3.0, so it's a great time to discuss should we make > Scala 2.12 as default Scala version in Spark 3.0. > > Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to > support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed > work per discussion in Scala community, > https://github.com/scala/scala-dev/issues/559#issuecomment-436160166 > > We have initial support of Scala 2.12 in Spark 2.4. If we decide to make > Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on > bugs and issues that we may run into. > > What do you think? > > Thanks, > > DB Tsai | Siri Open Source Technologies [not a contribution] | > Apple, Inc > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > <dev-unsubscr...@spark.apache.org> > > > -- Ryan Blue Software Engineer Netflix
