Re: Spark 2.3 in oozie
Wow, that's great news! Can I ask you to summarize the steps necessary to make this happen? It would be good to see everything together - also, it would probably help others as well. Thank you for sharing your struggles - and solutions as well! Peter On Wed, May 16, 2018 at 10:49 PM, purna pradeep <purna2prad...@gmail.com> wrote: > Thanks Peter! > > I’m able to run spark pi example on Kubernetes cluster from oozie after > this change > > On Wed, May 16, 2018 at 10:27 AM Peter Cseh <gezap...@cloudera.com> wrote: > > > The version of the xml schema has nothing to do with the version of the > > component you're using. > > > > Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required > > for compilation with Spark 2.3.0 > > > > Oozie does not pull in Spark's Kubernetes artifact. > > To make it part of the Oozie Spark sharelib you'll have to include the > > spark-kubernetes.jar > > < > > https://search.maven.org/#artifactdetails%7Corg.apache. > spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar > > > > > in > > the sharelib/spark/pom.xml as a compile-time dependency. > > > > gp > > > > On Tue, May 15, 2018 at 9:04 PM, purna pradeep <purna2prad...@gmail.com> > > wrote: > > > > > I’m able to compile successfully with after adding this override option > > > > > > -Dspark.scala.binary.version=2.11 > > > > > > Dspark.version = 2.3.0 > > > > > > But when I’m running a spark action with spark-pi example jar against > > > Kubernetes master I’m getting below error in stderr log > > > > > > > > > *Error:Could not load KUBERNETES classes.This copy of spark may not > have > > > been compiled with Kubernetes support* > > > > > > Below is my workflow.xml > > > > > > <*spark xmlns="uri:oozie:spark-action:1.0">* > > > > > > *${resourceManager}* > > > > > > *${nameNode}* > > > > > > *k8s://<***.com>* > > > > > > *Python-Spark-Pi* > > > > > > *spark-examples_2.11-2.3.0.jar* > > > > > > *--class org.apache.spark.examples.SparkPi > --conf > > > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark > --conf > > > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf > > > spark.kubernetes.container.image=artifactory.cloud. > > > capitalone.com/kubespark/spark-quantum:v2.3.0 > > > <http://artifactory.cloud.capitalone.com/kubespark/ > spark-quantum:v2.3.0> > > > --conf > > spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > > < > > http://spark.kubernetes.node.selector.node-role.kubernetes. > io/worker=true > > > > > > > --conf > > > spark.kubernetes.driver.label.application=is1-driver --conf > > > spark.kubernetes.executor.label.application=is1-exec*utor > > > local:///opt/spark/examples/jars/spark-examples_2.11-2.3. > > > 0.jar > > > > > > > > > > > > > > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it > > > needs to be spark-action:2.0 as I’m using spark 2.3? > > > > > > > > > Please suggest! > > > > > > > > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> > > wrote: > > > > > > > I think the error is related to the Scala version being present in > the > > > > artifact name. > > > > I'll take a look at this tomorrow. > > > > Gp > > > > > > > > On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> > > wrote: > > > > > > > > > Did you run > > > > > mvn clean install first on the parent directory? > > > > > > > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep < > > purna2prad...@gmail.com> > > > > > wrote: > > > > > > > > > > > Thanks peter, > > > > > > > > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie > > I’m > > > > > > getting below error from oozie examples > > > > > > > > > > > > > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could > not > > > > > resolve > > > > > > dependencies
Re: Spark 2.3 in oozie
Thanks Peter! I’m able to run spark pi example on Kubernetes cluster from oozie after this change On Wed, May 16, 2018 at 10:27 AM Peter Cseh <gezap...@cloudera.com> wrote: > The version of the xml schema has nothing to do with the version of the > component you're using. > > Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required > for compilation with Spark 2.3.0 > > Oozie does not pull in Spark's Kubernetes artifact. > To make it part of the Oozie Spark sharelib you'll have to include the > spark-kubernetes.jar > < > https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar > > > in > the sharelib/spark/pom.xml as a compile-time dependency. > > gp > > On Tue, May 15, 2018 at 9:04 PM, purna pradeep <purna2prad...@gmail.com> > wrote: > > > I’m able to compile successfully with after adding this override option > > > > -Dspark.scala.binary.version=2.11 > > > > Dspark.version = 2.3.0 > > > > But when I’m running a spark action with spark-pi example jar against > > Kubernetes master I’m getting below error in stderr log > > > > > > *Error:Could not load KUBERNETES classes.This copy of spark may not have > > been compiled with Kubernetes support* > > > > Below is my workflow.xml > > > > <*spark xmlns="uri:oozie:spark-action:1.0">* > > > > *${resourceManager}* > > > > *${nameNode}* > > > > *k8s://<***.com>* > > > > *Python-Spark-Pi* > > > > *spark-examples_2.11-2.3.0.jar* > > > > *--class org.apache.spark.examples.SparkPi --conf > > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf > > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf > > spark.kubernetes.container.image=artifactory.cloud. > > capitalone.com/kubespark/spark-quantum:v2.3.0 > > <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0> > > --conf > spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > < > http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > > > > --conf > > spark.kubernetes.driver.label.application=is1-driver --conf > > spark.kubernetes.executor.label.application=is1-exec*utor > > local:///opt/spark/examples/jars/spark-examples_2.11-2.3. > > 0.jar > > > > > > > > > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it > > needs to be spark-action:2.0 as I’m using spark 2.3? > > > > > > Please suggest! > > > > > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> > wrote: > > > > > I think the error is related to the Scala version being present in the > > > artifact name. > > > I'll take a look at this tomorrow. > > > Gp > > > > > > On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> > wrote: > > > > > > > Did you run > > > > mvn clean install first on the parent directory? > > > > > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep < > purna2prad...@gmail.com> > > > > wrote: > > > > > > > > > Thanks peter, > > > > > > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie > I’m > > > > > getting below error from oozie examples > > > > > > > > > > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not > > > > resolve > > > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: > > > Could > > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in > > > > resolution > > > > > * > > > > > > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com > > > > > > wrote: > > > > > > > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark > > > 2.1: > > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie > > that > > > > way > > > > > it > > > > > > will work. > > > > > > gp > > > > > > > > > > > > > > > > > > On Tue, May 15, 2018
Re: Spark 2.3 in oozie
The version of the xml schema has nothing to do with the version of the component you're using. Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required for compilation with Spark 2.3.0 Oozie does not pull in Spark's Kubernetes artifact. To make it part of the Oozie Spark sharelib you'll have to include the spark-kubernetes.jar <https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar> in the sharelib/spark/pom.xml as a compile-time dependency. gp On Tue, May 15, 2018 at 9:04 PM, purna pradeep <purna2prad...@gmail.com> wrote: > I’m able to compile successfully with after adding this override option > > -Dspark.scala.binary.version=2.11 > > Dspark.version = 2.3.0 > > But when I’m running a spark action with spark-pi example jar against > Kubernetes master I’m getting below error in stderr log > > > *Error:Could not load KUBERNETES classes.This copy of spark may not have > been compiled with Kubernetes support* > > Below is my workflow.xml > > <*spark xmlns="uri:oozie:spark-action:1.0">* > > *${resourceManager}* > > *${nameNode}* > > *k8s://<***.com>* > > *Python-Spark-Pi* > > *spark-examples_2.11-2.3.0.jar* > > *--class org.apache.spark.examples.SparkPi --conf > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf > spark.kubernetes.container.image=artifactory.cloud. > capitalone.com/kubespark/spark-quantum:v2.3.0 > <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0> > --conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > <http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > > --conf > spark.kubernetes.driver.label.application=is1-driver --conf > spark.kubernetes.executor.label.application=is1-exec*utor > local:///opt/spark/examples/jars/spark-examples_2.11-2.3. > 0.jar > > > > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it > needs to be spark-action:2.0 as I’m using spark 2.3? > > > Please suggest! > > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> wrote: > > > I think the error is related to the Scala version being present in the > > artifact name. > > I'll take a look at this tomorrow. > > Gp > > > > On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> wrote: > > > > > Did you run > > > mvn clean install first on the parent directory? > > > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com> > > > wrote: > > > > > > > Thanks peter, > > > > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > > > > getting below error from oozie examples > > > > > > > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not > > > resolve > > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: > > Could > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in > > > resolution > > > > * > > > > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com> > > > wrote: > > > > > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark > > 2.1: > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie > that > > > way > > > > it > > > > > will work. > > > > > gp > > > > > > > > > > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep < > > > purna2prad...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > Does oozie supports spark 2.3? Or will it even care of the spark > > > > version > > > > > > > > > > > > I want to use spark action > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Purna > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Peter Cseh *| Software Engineer > > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > [image: > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > > Cloudera > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > -- > > > > > > > > > > > > > > > -- *Peter Cseh *| Software Engineer cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> --
Re: Spark 2.3 in oozie
I’m able to compile successfully with after adding this override option -Dspark.scala.binary.version=2.11 Dspark.version = 2.3.0 But when I’m running a spark action with spark-pi example jar against Kubernetes master I’m getting below error in stderr log *Error:Could not load KUBERNETES classes.This copy of spark may not have been compiled with Kubernetes support* Below is my workflow.xml <*spark xmlns="uri:oozie:spark-action:1.0">* *${resourceManager}* *${nameNode}* *k8s://<***.com>* *Python-Spark-Pi* *spark-examples_2.11-2.3.0.jar* *--class org.apache.spark.examples.SparkPi --conf spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0 <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0> --conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true <http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true> --conf spark.kubernetes.driver.label.application=is1-driver --conf spark.kubernetes.executor.label.application=is1-exec*utor local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it needs to be spark-action:2.0 as I’m using spark 2.3? Please suggest! On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> wrote: > I think the error is related to the Scala version being present in the > artifact name. > I'll take a look at this tomorrow. > Gp > > On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> wrote: > > > Did you run > > mvn clean install first on the parent directory? > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com> > > wrote: > > > > > Thanks peter, > > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > > > getting below error from oozie examples > > > > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not > > resolve > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: > Could > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in > > resolution > > > * > > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com> > > wrote: > > > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark > 2.1: > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie that > > way > > > it > > > > will work. > > > > gp > > > > > > > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep < > > purna2prad...@gmail.com> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > Does oozie supports spark 2.3? Or will it even care of the spark > > > version > > > > > > > > > > I want to use spark action > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > Purna > > > > > > > > > > > > > > > > > > > > > -- > > > > *Peter Cseh *| Software Engineer > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > -- > > > > > > > > > >
Re: Spark 2.3 in oozie
I think the error is related to the Scala version being present in the artifact name. I'll take a look at this tomorrow. Gp On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> wrote: > Did you run > mvn clean install first on the parent directory? > > On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com> > wrote: > > > Thanks peter, > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > > getting below error from oozie examples > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not > resolve > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in > resolution > > * > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com> > wrote: > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > > I'm sure if you overwrite the -Dspark.version and compile Oozie that > way > > it > > > will work. > > > gp > > > > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep < > purna2prad...@gmail.com> > > > wrote: > > > > > > > Hello, > > > > > > > > Does oozie supports spark 2.3? Or will it even care of the spark > > version > > > > > > > > I want to use spark action > > > > > > > > > > > > > > > > Thanks, > > > > Purna > > > > > > > > > > > > > > > > -- > > > *Peter Cseh *| Software Engineer > > > cloudera.com <https://www.cloudera.com> > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > Cloudera > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > -- > > > > > >
Re: Spark 2.3 in oozie
Did you run mvn clean install first on the parent directory? On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com> wrote: > Thanks peter, > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m > getting below error from oozie examples > > > *ERROR] Failed to execute goal on project oozie-examples: Could not resolve > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution > * > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com> wrote: > > > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > I'm sure if you overwrite the -Dspark.version and compile Oozie that way > it > > will work. > > gp > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <purna2prad...@gmail.com> > > wrote: > > > > > Hello, > > > > > > Does oozie supports spark 2.3? Or will it even care of the spark > version > > > > > > I want to use spark action > > > > > > > > > > > > Thanks, > > > Purna > > > > > > > > > > > -- > > *Peter Cseh *| Software Engineer > > cloudera.com <https://www.cloudera.com> > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > -- > > >
Re: Spark 2.3 in oozie
Thanks peter, I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m getting below error from oozie examples *ERROR] Failed to execute goal on project oozie-examples: Could not resolve dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution * On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com> wrote: > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > I'm sure if you overwrite the -Dspark.version and compile Oozie that way it > will work. > gp > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <purna2prad...@gmail.com> > wrote: > > > Hello, > > > > Does oozie supports spark 2.3? Or will it even care of the spark version > > > > I want to use spark action > > > > > > > > Thanks, > > Purna > > > > > > -- > *Peter Cseh *| Software Engineer > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > -- >
Re: Spark 2.3 in oozie
Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1: https://github.com/apache/oozie/blob/master/pom.xml#L1983 I'm sure if you overwrite the -Dspark.version and compile Oozie that way it will work. gp On Tue, May 15, 2018 at 5:07 PM, purna pradeep <purna2prad...@gmail.com> wrote: > Hello, > > Does oozie supports spark 2.3? Or will it even care of the spark version > > I want to use spark action > > > > Thanks, > Purna > -- *Peter Cseh *| Software Engineer cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> --
Spark 2.3 in oozie
Hello, Does oozie supports spark 2.3? Or will it even care of the spark version I want to use spark action Thanks, Purna