Thanks Peter! I’m able to run spark pi example on Kubernetes cluster from oozie after this change
On Wed, May 16, 2018 at 10:27 AM Peter Cseh <gezap...@cloudera.com> wrote: > The version of the xml schema has nothing to do with the version of the > component you're using. > > Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required > for compilation with Spark 2.3.0 > > Oozie does not pull in Spark's Kubernetes artifact. > To make it part of the Oozie Spark sharelib you'll have to include the > spark-kubernetes.jar > < > https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar > > > in > the sharelib/spark/pom.xml as a compile-time dependency. > > gp > > On Tue, May 15, 2018 at 9:04 PM, purna pradeep <purna2prad...@gmail.com> > wrote: > > > I’m able to compile successfully with after adding this override option > > > > -Dspark.scala.binary.version=2.11 > > > > Dspark.version = 2.3.0 > > > > But when I’m running a spark action with spark-pi example jar against > > Kubernetes master I’m getting below error in stderr log > > > > > > *Error:Could not load KUBERNETES classes.This copy of spark may not have > > been compiled with Kubernetes support* > > > > Below is my workflow.xml > > > > <*spark xmlns="uri:oozie:spark-action:1.0">* > > > > * <resource-manager>${resourceManager}</resource-manager>* > > > > * <name-node>${nameNode}</name-node>* > > > > * <master>k8s://<***.com></master>* > > > > * <name>Python-Spark-Pi</name>* > > > > * <jar>spark-examples_2.11-2.3.0.jar</jar>* > > > > * <spark-opts>--class org.apache.spark.examples.SparkPi --conf > > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf > > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf > > spark.kubernetes.container.image=artifactory.cloud. > > capitalone.com/kubespark/spark-quantum:v2.3.0 > > <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0> > > --conf > spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > < > http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true > > > > > --conf > > spark.kubernetes.driver.label.application=is1-driver --conf > > spark.kubernetes.executor.label.application=is1-exec*utor > > local:///opt/spark/examples/jars/spark-examples_2.11-2.3. > > 0.jar</spark-opts> > > > > </spark> > > > > > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it > > needs to be spark-action:2.0 as I’m using spark 2.3? > > > > > > Please suggest! > > > > > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> > wrote: > > > > > I think the error is related to the Scala version being present in the > > > artifact name. > > > I'll take a look at this tomorrow. > > > Gp > > > > > > On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> > wrote: > > > > > > > Did you run > > > > mvn clean install first on the parent directory? > > > > > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep < > purna2prad...@gmail.com> > > > > wrote: > > > > > > > > > Thanks peter, > > > > > > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie > I’m > > > > > getting below error from oozie examples > > > > > > > > > > > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not > > > > resolve > > > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: > > > Could > > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in > > > > resolution > > > > > * > > > > > > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com > > > > > > wrote: > > > > > > > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark > > > 2.1: > > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983 > > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie > > that > > > > way > > > > > it > > > > > > will work. > > > > > > gp > > > > > > > > > > > > > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep < > > > > purna2prad...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > Does oozie supports spark 2.3? Or will it even care of the > spark > > > > > version > > > > > > > > > > > > > > I want to use spark action > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Purna > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Peter Cseh *| Software Engineer > > > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > [image: > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > [image: > > > > > Cloudera > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > -- > *Peter Cseh *| Software Engineer > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ >
