I’m able to compile successfully with after adding this override option
-Dspark.scala.binary.version=2.11
Dspark.version = 2.3.0
But when I’m running a spark action with spark-pi example jar against
Kubernetes master I’m getting below error in stderr log
*Error:Could not load KUBERNETES classes.This copy of spark may not have
been compiled with Kubernetes support*
Below is my workflow.xml
<*spark xmlns="uri:oozie:spark-action:1.0">*
* <resource-manager>${resourceManager}</resource-manager>*
* <name-node>${nameNode}</name-node>*
* <master>k8s://<***.com></master>*
* <name>Python-Spark-Pi</name>*
* <jar>spark-examples_2.11-2.3.0.jar</jar>*
* <spark-opts>--class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
spark.kubernetes.container.image=artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0
<http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0>
--conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
<http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true>
--conf
spark.kubernetes.driver.label.application=is1-driver --conf
spark.kubernetes.executor.label.application=is1-exec*utor
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar</spark-opts>
</spark>
Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
needs to be spark-action:2.0 as I’m using spark 2.3?
Please suggest!
On Tue, May 15, 2018 at 12:43 PM Peter Cseh <gezap...@cloudera.com> wrote:
> I think the error is related to the Scala version being present in the
> artifact name.
> I'll take a look at this tomorrow.
> Gp
>
> On Tue, May 15, 2018, 18:28 Artem Ervits <artemerv...@gmail.com> wrote:
>
> > Did you run
> > mvn clean install first on the parent directory?
> >
> > On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com>
> > wrote:
> >
> > > Thanks peter,
> > >
> > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > > getting below error from oozie examples
> > >
> > >
> > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > resolve
> > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> Could
> > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > resolution
> > > *
> > >
> > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezap...@cloudera.com>
> > wrote:
> > >
> > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> 2.1:
> > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > I'm sure if you overwrite the -Dspark.version and compile Oozie that
> > way
> > > it
> > > > will work.
> > > > gp
> > > >
> > > >
> > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > purna2prad...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > > version
> > > > >
> > > > > I want to use spark action
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Purna
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Peter Cseh *| Software Engineer
> > > > cloudera.com <https://www.cloudera.com>
> > > >
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > >
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > >
> > >
> >
>
