Re: Packages to release in 3.0.0-preview

On Thu, Oct 31, 2019 at 4:30 PM Sean Owen wrote: > > . But it'd be cooler to call these major > releases! Maybe this is just semantics, but my point is the Scala project already does call 2.12 to 2.13 a major release e.g. from https://www.scala-lang.org/download/ "Note that different *major*

Re: Packages to release in 3.0.0-preview

Yep, it's worse than that. Code compiled for 2.x is _not allowed_ to work with 2.(x+1). I say this with all love for Scala and total respect for how big improvements in what Scala does necessarily mean bytecode-level incompatibility. But it'd be cooler to call these major releases! even in Java,

Re: Packages to release in 3.0.0-preview

On Wed, Oct 30, 2019 at 5:57 PM Sean Owen wrote: > Or, frankly, maybe Scala should reconsider the mutual incompatibility > between minor releases. These are basically major releases, and > indeed, it causes exactly this kind of headache. > Not saying binary incompatibility is fun, but 2.12 to

Re: Packages to release in 3.0.0-preview

I don't agree with this take. The bottleneck is pretty much not Spark -- it is all of its dependencies, and there are unfortunately a lot. For example, Chill (among other things) doesn't support 2.13 yet. I don't think 2.13 is that 'mainstream' yet. We are not close to Scala 2.13 support, so it

Re: Packages to release in 3.0.0-preview

scala 2.13 support is tracked by https://issues.apache.org/jira/browse/SPARK-25075 , at the current time there are still major issues remaining, thus we don't include scala 2.13 support in the 3.0.0-preview release. If the task is finished before the code freeze of Spark 3.0.0, then it's still

Re: Packages to release in 3.0.0-preview

Why not trying the current Scala (2.13)? Spark has always been one (sometimes - two) Scala versions away from the whole Scala ecosystem and it has always been a big pain point for everybody. I understand that in the past you could not switch because of compatibility issues, but 3.x is a major

Re: Packages to release in 3.0.0-preview

Thank you Dongjoon. Please check out the latest code from test-spark-jdk11 . It works with JDK 1.8. One workaround is to install the Spark packages to local maven repository using hadoop-3.2 profile and JDK 1.8. On Mon, Oct 28, 2019 at 5:03 AM

Re: Packages to release in 3.0.0-preview

Hi, Yuming. Is the project working correctly on JDK8 with you? When I simply cloned your repo and did `mvn clean package` on JDK 1.8.0_232, it seems not to pass the UTs. I also tried to rerun after ignoring two ORC table test like the followings, but the UT is failing.

Re: Packages to release in 3.0.0-preview

It seems not a Hadoop issue, doesn't it? What Yuming pointed seems to be `Hive 2.3.6` profile implementation issue which is enabled only when `Hadoop 3.2`. >From my side, I'm +1 for publishing jars which depends on `Hadoop 3.2.0 / Hive 2.3.6` jars to Maven since Apache Spark 3.0.0. For the

Re: Packages to release in 3.0.0-preview

Is the Spark artifact actually any different between those builds? I thought it just affected what else was included in the binary tarball. If it matters, yes I'd publish a "Hadoop 3" version to Maven. (Scala 2.12 is the only supported Scala version). On Sun, Oct 27, 2019 at 4:35 AM Yuming Wang

Re: Packages to release in 3.0.0-preview

Do we need to publish the Scala 2.12 + hadoop 3.2 jar packages to the Maven repository? Otherwise it will throw a NoSuchMethodError on Java 11. Here is an example: https://github.com/wangyum/test-spark-jdk11/blob/master/src/test/scala/test/spark/HiveTableSuite.scala#L34-L38

Re: Packages to release in 3.0.0-preview

Thanks for that work! > I don't think JDK 11 is a separate release (by design). We build > everything targeting JDK 8 and it should work on JDK 11 too. +1. a single package working on both jvms looks nice. On Sat, Oct 26, 2019 at 4:18 AM Sean Owen wrote: > I don't think JDK 11 is a separate

Re: Packages to release in 3.0.0-preview

I don't think JDK 11 is a separate release (by design). We build everything targeting JDK 8 and it should work on JDK 11 too. So, just two releases, but, frankly I think we soon need to stop multiple releases for multiple Hadoop versions, and stick to Hadoop 3. I think it's fine to try to release

Packages to release in 3.0.0-preview

Hi all, I would like to bring out a discussion on how many packages shall be released in 3.0.0-preview, the ones I can think of now: * scala 2.12 + hadoop 2.7 * scala 2.12 + hadoop 3.2 * scala 2.12 + hadoop 3.2 + JDK 11 Do you have other combinations to add to the above list? Cheers, Xingbo