Re: Spark 2.3 in oozie

I’m able to compile successfully with after adding this override option -Dspark.scala.binary.version=2.11 Dspark.version = 2.3.0 But when I’m running a spark action with spark-pi example jar against Kubernetes master I’m getting below error in stderr log *Error:Could not load KUBERNETES

Re: Spark 2.3 in oozie

I think the error is related to the Scala version being present in the artifact name. I'll take a look at this tomorrow. Gp On Tue, May 15, 2018, 18:28 Artem Ervits wrote: > Did you run > mvn clean install first on the parent directory? > > On Tue, May 15, 2018, 11:35 AM

Re: Spark 2.3 in oozie

Did you run mvn clean install first on the parent directory? On Tue, May 15, 2018, 11:35 AM purna pradeep wrote: > Thanks peter, > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > getting below error from oozie examples > > > *ERROR] Failed to

Re: Spark 2.3 in oozie

Thanks peter, I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m getting below error from oozie examples *ERROR] Failed to execute goal on project oozie-examples: Could not resolve dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could not find artifact

Re: Spark 2.3 in oozie

Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: https://github.com/apache/oozie/blob/master/pom.xml#L1983 I'm sure if you overwrite the -Dspark.version and compile Oozie that way it will work. gp On Tue, May 15, 2018 at 5:07 PM, purna pradeep

Spark 2.3 in oozie

Hello, Does oozie supports spark 2.3? Or will it even care of the spark version I want to use spark action Thanks, Purna

Re: Oozie for spark jobs without Hadoop

Thanks Andras, Also I also would like to know if oozie supports Aws S3 as input events to poll for a dependency file before kicking off a spark action For example: I don’t want to kick off a spark action until a file is arrived on a given AWS s3 location On Tue, May 15, 2018 at 10:17 AM Andras

Re: Oozie for spark jobs without Hadoop

Hi, Oozie needs HDFS to store workflow, coordinator, or bundle definitions, as well as sharelib files in a safe, distributed and scalable way. Oozie needs YARN to run almost all of its actions, Spark action being no exception. At the moment it's not feasible to install Oozie without those Hadoop