Re: Apache Spark 3.5.0 Expectations (?)

2023-05-29 Thread Jungtaek Lim
Shall we initiate a new discussion thread for Scala 2.13 by default? While I'm not an expert on this area, it sounds like the change is major and (probably) breaking. It seems to be worth having a separate discussion thread rather than just treat it like one of 25 items. On Tue, May 30, 2023 at

Re: Apache Spark 3.5.0 Expectations (?)

2023-05-29 Thread Sean Owen
It does seem risky; there are still likely libs out there that don't cross compile for 2.13. I would make it the default at 4.0, myself. On Mon, May 29, 2023 at 7:16 PM Hyukjin Kwon wrote: > While I support going forward with a higher version, actually using Scala > 2.13 by default is a big

Re: Apache Spark 3.5.0 Expectations (?)

2023-05-29 Thread Hyukjin Kwon
While I support going forward with a higher version, actually using Scala 2.13 by default is a big deal especially in a way that: - Users would likely download the built-in version assuming that it’s backward binary compatible. - PyPI doesn't allow specifying the Scala version, meaning