Re: Spark 2.3 in oozie

Wow, that's great news! Can I ask you to summarize the steps necessary to make this happen? It would be good to see everything together - also, it would probably help others as well. Thank you for sharing your struggles - and solutions as well! Peter On Wed, May 16, 2018 at 10:49 PM, purna

Re: Spark 2.3 in oozie

Thanks Peter! I’m able to run spark pi example on Kubernetes cluster from oozie after this change On Wed, May 16, 2018 at 10:27 AM Peter Cseh wrote: > The version of the xml schema has nothing to do with the version of the > component you're using. > > Thanks for

Re: Spark 2.3 in oozie

The version of the xml schema has nothing to do with the version of the component you're using. Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required for compilation with Spark 2.3.0 Oozie does not pull in Spark's Kubernetes artifact. To make it part of the Oozie Spark

Re: Spark 2.3 in oozie

I’m able to compile successfully with after adding this override option -Dspark.scala.binary.version=2.11 Dspark.version = 2.3.0 But when I’m running a spark action with spark-pi example jar against Kubernetes master I’m getting below error in stderr log *Error:Could not load KUBERNETES

Re: Spark 2.3 in oozie

I think the error is related to the Scala version being present in the artifact name. I'll take a look at this tomorrow. Gp On Tue, May 15, 2018, 18:28 Artem Ervits wrote: > Did you run > mvn clean install first on the parent directory? > > On Tue, May 15, 2018, 11:35 AM

Re: Spark 2.3 in oozie

Did you run mvn clean install first on the parent directory? On Tue, May 15, 2018, 11:35 AM purna pradeep wrote: > Thanks peter, > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > getting below error from oozie examples > > > *ERROR] Failed to

Re: Spark 2.3 in oozie

Thanks peter, I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m getting below error from oozie examples *ERROR] Failed to execute goal on project oozie-examples: Could not resolve dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could not find artifact

Re: Spark 2.3 in oozie

Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: https://github.com/apache/oozie/blob/master/pom.xml#L1983 I'm sure if you overwrite the -Dspark.version and compile Oozie that way it will work. gp On Tue, May 15, 2018 at 5:07 PM, purna pradeep