Thanks for your response. I think I misinterpreted the stability/compatibility guarantee with 1.0 release. It seems like the compatibility is only at the API level.
This is interesting because it means any system/product that is built on top of Spark and uses Spark with a long-running SparkContext connecting to the cluster over network, will need to make sure it has the exact same version of Spark jar as the cluster, even to the patch version. This would be analogous to having to compile Spark against a very specific version of Hadoop, as opposed to currently being able to use the Spark package with CDH4 against most of the CDH4 Hadoop clusters. Is it correct that Spark is focusing and prioritizing around the spark-submit use cases than the aforementioned use cases? I just wanted to better understand the future direction/prioritization of spark. Thanks, Mingyu From: Patrick Wendell <pwend...@gmail.com> Date: Thursday, August 14, 2014 at 6:32 PM To: Gary Malouf <malouf.g...@gmail.com> Cc: Mingyu Kim <m...@palantir.com>, "dev@spark.apache.org" <dev@spark.apache.org> Subject: Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster I commented on the bug. For driver mode, you'll need to get the corresponding version of spark-submit for Spark 1.0.2. On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf <malouf.g...@gmail.com> wrote: > To be clear, is it 'compiled' against 1.0.2 or it packaged with it? > > > On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim <m...@palantir.com> wrote: > >> > I ran a really simple code that runs with Spark 1.0.2 jar and connects to >> > a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I >> > filed the bug at https://issues.apache.org/jira/browse/SPARK-3050 >> <https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/br >> owse/SPARK-3050&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=UKDOcu6qL3KsoZhpOohNBR1uc >> PNmWnbd3eEJ9hVUdMk%3D%0A&m=qvQ59wZwD7EuezjTuLzmNTRUamDRDnI7%2F0%2BnULtXk4k%3D >> %0A&s=b7abf7638a3e6fac2ddac9d8f0ca52f1a92945465abfb2e2d996a96d2301fec5> . >> > >> > I assumed the minor and patch releases shouldn¹t break compatibility. Is >> > that correct? >> > >> > Thanks, >> > Mingyu >> >
smime.p7s
Description: S/MIME cryptographic signature
