Re: Spark 2.3 in oozie

[Peter Cseh]

Wow, that's great news!

Can I ask you to summarize the steps necessary to make this happen? It
would be good to see everything together - also, it would probably help
others as well.

Thank you for sharing your struggles - and solutions as well!

Peter

On Wed, May 16, 2018 at 10:49 PM, purna pradeep 
wrote:

> Thanks Peter!
>
> I’m able to run spark pi example on Kubernetes cluster from oozie after
> this change
>
> On Wed, May 16, 2018 at 10:27 AM Peter Cseh  wrote:
>
> > The version of the xml schema has nothing to do with the version of the
> > component you're using.
> >
> > Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
> > for compilation with Spark 2.3.0
> >
> > Oozie does not pull in Spark's Kubernetes artifact.
> > To make it part of the Oozie Spark sharelib you'll have to include the
> > spark-kubernetes.jar
> > <
> > https://search.maven.org/#artifactdetails%7Corg.apache.
> spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar
> > >
> > in
> > the sharelib/spark/pom.xml as a compile-time dependency.
> >
> > gp
> >
> > On Tue, May 15, 2018 at 9:04 PM, purna pradeep 
> > wrote:
> >
> > > I’m able to compile successfully with after adding this override option
> > >
> > > -Dspark.scala.binary.version=2.11
> > >
> > > Dspark.version = 2.3.0
> > >
> > > But when I’m running a spark action with spark-pi example jar against
> > > Kubernetes master I’m getting below error in stderr log
> > >
> > >
> > > *Error:Could not load KUBERNETES classes.This copy of spark may not
> have
> > > been compiled with Kubernetes support*
> > >
> > > Below is my workflow.xml
> > >
> > > <*spark xmlns="uri:oozie:spark-action:1.0">*
> > >
> > > *${resourceManager}*
> > >
> > > *${nameNode}*
> > >
> > > *k8s://<***.com>*
> > >
> > > *Python-Spark-Pi*
> > >
> > > *spark-examples_2.11-2.3.0.jar*
> > >
> > > *--class org.apache.spark.examples.SparkPi
> --conf
> > > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark
> --conf
> > > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> > > spark.kubernetes.container.image=artifactory.cloud.
> > > capitalone.com/kubespark/spark-quantum:v2.3.0
> > >  spark-quantum:v2.3.0>
> > > --conf
> > spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > > <
> > http://spark.kubernetes.node.selector.node-role.kubernetes.
> io/worker=true
> > > >
> > > --conf
> > > spark.kubernetes.driver.label.application=is1-driver --conf
> > > spark.kubernetes.executor.label.application=is1-exec*utor
> > > local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> > > 0.jar
> > >
> > > 
> > >
> > >
> > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> > > needs to be spark-action:2.0 as I’m using spark 2.3?
> > >
> > >
> > > Please suggest!
> > >
> > >
> > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh 
> > wrote:
> > >
> > > > I think the error is related to the Scala version being present in
> the
> > > > artifact name.
> > > > I'll take a look at this tomorrow.
> > > > Gp
> > > >
> > > > On Tue, May 15, 2018, 18:28 Artem Ervits 
> > wrote:
> > > >
> > > > > Did you run
> > > > > mvn clean install first on the parent directory?
> > > > >
> > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <
> > purna2prad...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks peter,
> > > > > >
> > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie
> > I’m
> > > > > > getting below error from oozie examples
> > > > > >
> > > > > >
> > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could
> not
> > > > > resolve
> > > > > > dependencies for project org.apache.oozie:oozie-
> examples:jar:5.0.0:
> > > > Could
> > > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > > > resolution
> > > > > > *
> > > > > >
> > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <
> gezap...@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Oozie has a spark-2 profile that is currently hard-coded to
> Spark
> > > > 2.1:
> > > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> > > that
> > > > > way
> > > > > > it
> > > > > > > will work.
> > > > > > > gp
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > > > purna2prad...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > Does oozie supports spark 2.3? Or will it even care of the
> > spark
> > > > > > version
> > > > > > > >
> > > > > > > > I want to use spark action
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Purna
> > > > > > > >
> > > > > 

Re: Spark 2.3 in oozie

[purna pradeep]

Thanks Peter!

I’m able to run spark pi example on Kubernetes cluster from oozie after
this change

On Wed, May 16, 2018 at 10:27 AM Peter Cseh  wrote:

> The version of the xml schema has nothing to do with the version of the
> component you're using.
>
> Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
> for compilation with Spark 2.3.0
>
> Oozie does not pull in Spark's Kubernetes artifact.
> To make it part of the Oozie Spark sharelib you'll have to include the
> spark-kubernetes.jar
> <
> https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar
> >
> in
> the sharelib/spark/pom.xml as a compile-time dependency.
>
> gp
>
> On Tue, May 15, 2018 at 9:04 PM, purna pradeep 
> wrote:
>
> > I’m able to compile successfully with after adding this override option
> >
> > -Dspark.scala.binary.version=2.11
> >
> > Dspark.version = 2.3.0
> >
> > But when I’m running a spark action with spark-pi example jar against
> > Kubernetes master I’m getting below error in stderr log
> >
> >
> > *Error:Could not load KUBERNETES classes.This copy of spark may not have
> > been compiled with Kubernetes support*
> >
> > Below is my workflow.xml
> >
> > <*spark xmlns="uri:oozie:spark-action:1.0">*
> >
> > *${resourceManager}*
> >
> > *${nameNode}*
> >
> > *k8s://<***.com>*
> >
> > *Python-Spark-Pi*
> >
> > *spark-examples_2.11-2.3.0.jar*
> >
> > *--class org.apache.spark.examples.SparkPi --conf
> > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
> > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> > spark.kubernetes.container.image=artifactory.cloud.
> > capitalone.com/kubespark/spark-quantum:v2.3.0
> > 
> > --conf
> spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > <
> http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > >
> > --conf
> > spark.kubernetes.driver.label.application=is1-driver --conf
> > spark.kubernetes.executor.label.application=is1-exec*utor
> > local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> > 0.jar
> >
> > 
> >
> >
> > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> > needs to be spark-action:2.0 as I’m using spark 2.3?
> >
> >
> > Please suggest!
> >
> >
> > On Tue, May 15, 2018 at 12:43 PM Peter Cseh 
> wrote:
> >
> > > I think the error is related to the Scala version being present in the
> > > artifact name.
> > > I'll take a look at this tomorrow.
> > > Gp
> > >
> > > On Tue, May 15, 2018, 18:28 Artem Ervits 
> wrote:
> > >
> > > > Did you run
> > > > mvn clean install first on the parent directory?
> > > >
> > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <
> purna2prad...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks peter,
> > > > >
> > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie
> I’m
> > > > > getting below error from oozie examples
> > > > >
> > > > >
> > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > > > resolve
> > > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> > > Could
> > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > > resolution
> > > > > *
> > > > >
> > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh  >
> > > > wrote:
> > > > >
> > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> > > 2.1:
> > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> > that
> > > > way
> > > > > it
> > > > > > will work.
> > > > > > gp
> > > > > >
> > > > > >
> > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > > purna2prad...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Does oozie supports spark 2.3? Or will it even care of the
> spark
> > > > > version
> > > > > > >
> > > > > > > I want to use spark action
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Purna
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Peter Cseh *| Software Engineer
> > > > > > cloudera.com 
> > > > > >
> > > > > > [image: Cloudera] 
> > > > > >
> > > > > > [image: Cloudera on Twitter] 
> > [image:
> > > > > > Cloudera on Facebook] 
> [image:
> > > > > Cloudera
> > > > > > on LinkedIn] 
> > > > > > --
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com 
>
> 

Re: Oozie for spark jobs without Hadoop

[Peter Cseh]

Great progress there purna! :)

Have you tried adding these properites to the coordinator's configuration?
we usually use the action config to build up connection to the distributed
file system.
Although I'm not sure we're using these when polling the dependencies for
coordinators, but I'm excited about you trying to make it work!

I'll get back with a - hopefully - more helpful answer soon, I have to
check the code in more depth first.
gp

On Wed, May 16, 2018 at 9:45 PM, purna pradeep 
wrote:

> Peter,
>
> I got rid of this error by adding
> hadoop-aws-2.8.3.jar and jets3t-0.9.4.jar
>
> But I’m getting below error now
>
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
> Key must be specified by setting the fs.s3.awsAccessKeyId and
> fs.s3.awsSecretAccessKey properties (respectively)
>
> I have tried adding AWS access ,secret keys in
>
> oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml
>
>
>
>
> On Wed, May 16, 2018 at 2:30 PM purna pradeep 
> wrote:
>
>>
>> I have tried this ,just added s3 instead of *
>>
>> 
>>
>> oozie.service.HadoopAccessorService.
>> supported.filesystems
>>
>> hdfs,hftp,webhdfs,s3
>>
>> 
>>
>>
>> Getting below error
>>
>> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>> org.apache.hadoop.fs.s3a.S3AFileSystem not found
>>
>> at org.apache.hadoop.conf.Configuration.getClass(
>> Configuration.java:2369)
>>
>> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(
>> FileSystem.java:2793)
>>
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(
>> FileSystem.java:2810)
>>
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
>>
>> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
>> FileSystem.java:2849)
>>
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
>>
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
>>
>> at org.apache.oozie.service.HadoopAccessorService$5.run(
>> HadoopAccessorService.java:625)
>>
>> at org.apache.oozie.service.HadoopAccessorService$5.run(
>> HadoopAccessorService.java:623
>>
>>
>> On Wed, May 16, 2018 at 2:19 PM purna pradeep 
>> wrote:
>>
>>> This is what is in the logs
>>>
>>> 2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
>>> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
>>>
>>> 2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
>>> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
>>>
>>>
>>> On Wed, May 16, 2018 at 12:27 PM Peter Cseh 
>>> wrote:
>>>
 That's strange, this exception should not happen in that case.
 Can you check the server logs for messages like this?
 LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
 LOG.info("Loaded default urihandler {0}",
 defaultHandler.getClass().getName());
 Thanks

 On Wed, May 16, 2018 at 5:47 PM, purna pradeep  wrote:

> This is what I already have in my oozie-site.xml
>
> 
>
> oozie.service.HadoopAccessorService.
> supported.filesystems
>
> *
>
> 
>
> On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
> wrote:
>
>> You'll have to configure
>> oozie.service.HadoopAccessorService.supported.filesystems
>> hdfs,hftp,webhdfs Enlist
>> the different filesystems supported for federation. If wildcard "*" is
>> specified, then ALL file schemes will be allowed.properly.
>>
>> For testing purposes it's ok to put * in there in oozie-site.xml
>>
>> On Wed, May 16, 2018 at 5:29 PM, purna pradeep <
>> purna2prad...@gmail.com>
>> wrote:
>>
>> > Peter,
>> >
>> > I have tried to specify dataset with uri starting with s3://,
>> s3a:// and
>> > s3n:// and I am getting exception
>> >
>> >
>> >
>> > Exception occurred:E0904: Scheme [s3] not supported in uri
>> > [s3://mybucket/input.data] Making the job failed
>> >
>> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme
>> [s3] not
>> > supported in uri [s3:// mybucket /input.data]
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:185)
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:168)
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:160)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
>> > CoordCommandUtils.java:465)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.
>> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>> >

Re: Oozie for spark jobs without Hadoop

[Artem Ervits]

Here's some related info

https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-trouble/index.html

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md


On Wed, May 16, 2018, 3:45 PM purna pradeep  wrote:

> Peter,
>
> I got rid of this error by adding
> hadoop-aws-2.8.3.jar and jets3t-0.9.4.jar
>
> But I’m getting below error now
>
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
> must be specified by setting the fs.s3.awsAccessKeyId and
> fs.s3.awsSecretAccessKey properties (respectively)
>
> I have tried adding AWS access ,secret keys in
>
> oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml
>
>
>
>
> On Wed, May 16, 2018 at 2:30 PM purna pradeep 
> wrote:
>
> >
> > I have tried this ,just added s3 instead of *
> >
> > 
> >
> >
>  oozie.service.HadoopAccessorService.supported.filesystems
> >
> > hdfs,hftp,webhdfs,s3
> >
> > 
> >
> >
> > Getting below error
> >
> > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> > org.apache.hadoop.fs.s3a.S3AFileSystem not found
> >
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)
> >
> > at
> > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)
> >
> > at
> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> >
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> >
> > at
> > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> >
> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> >
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> >
> > at
> >
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
> >
> > at
> >
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623
> >
> >
> > On Wed, May 16, 2018 at 2:19 PM purna pradeep 
> > wrote:
> >
> >> This is what is in the logs
> >>
> >> 2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
> >> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
> >>
> >> 2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
> >> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
> >>
> >>
> >> On Wed, May 16, 2018 at 12:27 PM Peter Cseh 
> >> wrote:
> >>
> >>> That's strange, this exception should not happen in that case.
> >>> Can you check the server logs for messages like this?
> >>> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
> >>> LOG.info("Loaded default urihandler {0}",
> >>> defaultHandler.getClass().getName());
> >>> Thanks
> >>>
> >>> On Wed, May 16, 2018 at 5:47 PM, purna pradeep <
> purna2prad...@gmail.com>
> >>> wrote:
> >>>
>  This is what I already have in my oozie-site.xml
> 
>  
> 
> 
>  oozie.service.HadoopAccessorService.supported.filesystems
> 
>  *
> 
>  
> 
>  On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
>  wrote:
> 
> > You'll have to configure
> > oozie.service.HadoopAccessorService.supported.filesystems
> > hdfs,hftp,webhdfs Enlist
> > the different filesystems supported for federation. If wildcard "*"
> is
> > specified, then ALL file schemes will be allowed.properly.
> >
> > For testing purposes it's ok to put * in there in oozie-site.xml
> >
> > On Wed, May 16, 2018 at 5:29 PM, purna pradeep <
> > purna2prad...@gmail.com>
> > wrote:
> >
> > > Peter,
> > >
> > > I have tried to specify dataset with uri starting with s3://,
> s3a://
> > and
> > > s3n:// and I am getting exception
> > >
> > >
> > >
> > > Exception occurred:E0904: Scheme [s3] not supported in uri
> > > [s3://mybucket/input.data] Making the job failed
> > >
> > > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3]
> > not
> > > supported in uri [s3:// mybucket /input.data]
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:185)
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:168)
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:160)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
> > > CoordCommandUtils.java:465)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.
> > > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

Peter,

I got rid of this error by adding
hadoop-aws-2.8.3.jar and jets3t-0.9.4.jar

But I’m getting below error now

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
must be specified by setting the fs.s3.awsAccessKeyId and
fs.s3.awsSecretAccessKey properties (respectively)

I have tried adding AWS access ,secret keys in

oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml




On Wed, May 16, 2018 at 2:30 PM purna pradeep 
wrote:

>
> I have tried this ,just added s3 instead of *
>
> 
>
> oozie.service.HadoopAccessorService.supported.filesystems
>
> hdfs,hftp,webhdfs,s3
>
> 
>
>
> Getting below error
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3a.S3AFileSystem not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)
>
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)
>
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
>
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
>
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
>
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
>
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
>
> at
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
>
> at
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623
>
>
> On Wed, May 16, 2018 at 2:19 PM purna pradeep 
> wrote:
>
>> This is what is in the logs
>>
>> 2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
>> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
>>
>> 2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
>> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
>>
>>
>> On Wed, May 16, 2018 at 12:27 PM Peter Cseh 
>> wrote:
>>
>>> That's strange, this exception should not happen in that case.
>>> Can you check the server logs for messages like this?
>>> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
>>> LOG.info("Loaded default urihandler {0}",
>>> defaultHandler.getClass().getName());
>>> Thanks
>>>
>>> On Wed, May 16, 2018 at 5:47 PM, purna pradeep 
>>> wrote:
>>>
 This is what I already have in my oozie-site.xml

 


 oozie.service.HadoopAccessorService.supported.filesystems

 *

 

 On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
 wrote:

> You'll have to configure
> oozie.service.HadoopAccessorService.supported.filesystems
> hdfs,hftp,webhdfs Enlist
> the different filesystems supported for federation. If wildcard "*" is
> specified, then ALL file schemes will be allowed.properly.
>
> For testing purposes it's ok to put * in there in oozie-site.xml
>
> On Wed, May 16, 2018 at 5:29 PM, purna pradeep <
> purna2prad...@gmail.com>
> wrote:
>
> > Peter,
> >
> > I have tried to specify dataset with uri starting with s3://, s3a://
> and
> > s3n:// and I am getting exception
> >
> >
> >
> > Exception occurred:E0904: Scheme [s3] not supported in uri
> > [s3://mybucket/input.data] Making the job failed
> >
> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3]
> not
> > supported in uri [s3:// mybucket /input.data]
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:185)
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:168)
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:160)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
> > CoordCommandUtils.java:465)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.
> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.
> > materializeInputDataEvents(CoordCommandUtils.java:731)
> >
> > at
> >
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
> > CoordCommandUtils.java:546)
> >
> > at
> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
> >
> > at
> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
> >
> > at
> > 

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

I have tried this ,just added s3 instead of *



oozie.service.HadoopAccessorService.supported.filesystems

hdfs,hftp,webhdfs,s3




Getting below error

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.s3a.S3AFileSystem not found

at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)

at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)

at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)

at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)

at
org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)

at
org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623


On Wed, May 16, 2018 at 2:19 PM purna pradeep 
wrote:

> This is what is in the logs
>
> 2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
>
> 2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
>
>
> On Wed, May 16, 2018 at 12:27 PM Peter Cseh  wrote:
>
>> That's strange, this exception should not happen in that case.
>> Can you check the server logs for messages like this?
>> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
>> LOG.info("Loaded default urihandler {0}",
>> defaultHandler.getClass().getName());
>> Thanks
>>
>> On Wed, May 16, 2018 at 5:47 PM, purna pradeep 
>> wrote:
>>
>>> This is what I already have in my oozie-site.xml
>>>
>>> 
>>>
>>>
>>> oozie.service.HadoopAccessorService.supported.filesystems
>>>
>>> *
>>>
>>> 
>>>
>>> On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
>>> wrote:
>>>
 You'll have to configure
 oozie.service.HadoopAccessorService.supported.filesystems
 hdfs,hftp,webhdfs Enlist
 the different filesystems supported for federation. If wildcard "*" is
 specified, then ALL file schemes will be allowed.properly.

 For testing purposes it's ok to put * in there in oozie-site.xml

 On Wed, May 16, 2018 at 5:29 PM, purna pradeep 
 wrote:

 > Peter,
 >
 > I have tried to specify dataset with uri starting with s3://, s3a://
 and
 > s3n:// and I am getting exception
 >
 >
 >
 > Exception occurred:E0904: Scheme [s3] not supported in uri
 > [s3://mybucket/input.data] Making the job failed
 >
 > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3]
 not
 > supported in uri [s3:// mybucket /input.data]
 >
 > at
 > org.apache.oozie.service.URIHandlerService.getURIHandler(
 > URIHandlerService.java:185)
 >
 > at
 > org.apache.oozie.service.URIHandlerService.getURIHandler(
 > URIHandlerService.java:168)
 >
 > at
 > org.apache.oozie.service.URIHandlerService.getURIHandler(
 > URIHandlerService.java:160)
 >
 > at
 > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
 > CoordCommandUtils.java:465)
 >
 > at
 > org.apache.oozie.command.coord.CoordCommandUtils.
 > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
 >
 > at
 > org.apache.oozie.command.coord.CoordCommandUtils.
 > materializeInputDataEvents(CoordCommandUtils.java:731)
 >
 > at
 >
 org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
 > CoordCommandUtils.java:546)
 >
 > at
 > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
 > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
 >
 > at
 > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
 > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
 >
 > at
 > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
 > MaterializeTransitionXCommand.java:73)
 >
 > at
 > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
 > MaterializeTransitionXCommand.java:29)
 >
 > at org.apache.oozie.command.XCommand.call(XCommand.java:290)
 >
 > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 >
 > at
 > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
 > CallableQueueService.java:181)
 >
 > at
 > java.util.concurrent.ThreadPoolExecutor.runWorker(
 > ThreadPoolExecutor.java:1149)
 >
 > at
 > java.util.concurrent.ThreadPoolExecutor$Worker.run(

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

This is what is in the logs

2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]

2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
Loaded default urihandler org.apache.oozie.dependency.FSURIHandler


On Wed, May 16, 2018 at 12:27 PM Peter Cseh  wrote:

> That's strange, this exception should not happen in that case.
> Can you check the server logs for messages like this?
> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
> LOG.info("Loaded default urihandler {0}",
> defaultHandler.getClass().getName());
> Thanks
>
> On Wed, May 16, 2018 at 5:47 PM, purna pradeep 
> wrote:
>
>> This is what I already have in my oozie-site.xml
>>
>> 
>>
>>
>> oozie.service.HadoopAccessorService.supported.filesystems
>>
>> *
>>
>> 
>>
>> On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
>> wrote:
>>
>>> You'll have to configure
>>> oozie.service.HadoopAccessorService.supported.filesystems
>>> hdfs,hftp,webhdfs Enlist
>>> the different filesystems supported for federation. If wildcard "*" is
>>> specified, then ALL file schemes will be allowed.properly.
>>>
>>> For testing purposes it's ok to put * in there in oozie-site.xml
>>>
>>> On Wed, May 16, 2018 at 5:29 PM, purna pradeep 
>>> wrote:
>>>
>>> > Peter,
>>> >
>>> > I have tried to specify dataset with uri starting with s3://, s3a://
>>> and
>>> > s3n:// and I am getting exception
>>> >
>>> >
>>> >
>>> > Exception occurred:E0904: Scheme [s3] not supported in uri
>>> > [s3://mybucket/input.data] Making the job failed
>>> >
>>> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
>>> > supported in uri [s3:// mybucket /input.data]
>>> >
>>> > at
>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>> > URIHandlerService.java:185)
>>> >
>>> > at
>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>> > URIHandlerService.java:168)
>>> >
>>> > at
>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>> > URIHandlerService.java:160)
>>> >
>>> > at
>>> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
>>> > CoordCommandUtils.java:465)
>>> >
>>> > at
>>> > org.apache.oozie.command.coord.CoordCommandUtils.
>>> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>>> >
>>> > at
>>> > org.apache.oozie.command.coord.CoordCommandUtils.
>>> > materializeInputDataEvents(CoordCommandUtils.java:731)
>>> >
>>> > at
>>> >
>>> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
>>> > CoordCommandUtils.java:546)
>>> >
>>> > at
>>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>>> > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
>>> >
>>> > at
>>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>>> > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
>>> >
>>> > at
>>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>>> > MaterializeTransitionXCommand.java:73)
>>> >
>>> > at
>>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>>> > MaterializeTransitionXCommand.java:29)
>>> >
>>> > at org.apache.oozie.command.XCommand.call(XCommand.java:290)
>>> >
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> >
>>> > at
>>> > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
>>> > CallableQueueService.java:181)
>>> >
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> > ThreadPoolExecutor.java:1149)
>>> >
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> > ThreadPoolExecutor.java:624)
>>> >
>>> > at java.lang.Thread.run(Thread.java:748)
>>> >
>>> >
>>> >
>>> > Is S3 support specific to CDH distribution or should it work in Apache
>>> > Oozie as well? I’m not using CDH yet so
>>> >
>>> > On Wed, May 16, 2018 at 10:28 AM Peter Cseh 
>>> wrote:
>>> >
>>> > > I think it should be possible for Oozie to poll S3. Check out this
>>> > > <
>>> > > https://www.cloudera.com/documentation/enterprise/5-9-
>>> > x/topics/admin_oozie_s3.html
>>> > > >
>>> > > description on how to make it work in jobs, something similar should
>>> work
>>> > > on the server side as well
>>> > >
>>> > > On Tue, May 15, 2018 at 4:43 PM, purna pradeep <
>>> purna2prad...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Thanks Andras,
>>> > > >
>>> > > > Also I also would like to know if oozie supports Aws S3 as input
>>> events
>>> > > to
>>> > > > poll for a dependency file before kicking off a spark action
>>> > > >
>>> > > >
>>> > > > For example: I don’t want to kick off a spark action until a file
>>> is
>>> > > > arrived on a given AWS s3 location
>>> > > >
>>> > > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
>>> > andras.pi...@cloudera.com

Re: Oozie for spark jobs without Hadoop

[Peter Cseh]

That's strange, this exception should not happen in that case.
Can you check the server logs for messages like this?
LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
LOG.info("Loaded default urihandler {0}",
defaultHandler.getClass().getName());
Thanks

On Wed, May 16, 2018 at 5:47 PM, purna pradeep 
wrote:

> This is what I already have in my oozie-site.xml
>
> 
>
> oozie.service.HadoopAccessorService.
> supported.filesystems
>
> *
>
> 
>
> On Wed, May 16, 2018 at 11:37 AM Peter Cseh  wrote:
>
>> You'll have to configure
>> oozie.service.HadoopAccessorService.supported.filesystems
>> hdfs,hftp,webhdfs Enlist
>> the different filesystems supported for federation. If wildcard "*" is
>> specified, then ALL file schemes will be allowed.properly.
>>
>> For testing purposes it's ok to put * in there in oozie-site.xml
>>
>> On Wed, May 16, 2018 at 5:29 PM, purna pradeep 
>> wrote:
>>
>> > Peter,
>> >
>> > I have tried to specify dataset with uri starting with s3://, s3a:// and
>> > s3n:// and I am getting exception
>> >
>> >
>> >
>> > Exception occurred:E0904: Scheme [s3] not supported in uri
>> > [s3://mybucket/input.data] Making the job failed
>> >
>> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
>> > supported in uri [s3:// mybucket /input.data]
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:185)
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:168)
>> >
>> > at
>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>> > URIHandlerService.java:160)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
>> > CoordCommandUtils.java:465)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.
>> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.
>> > materializeInputDataEvents(CoordCommandUtils.java:731)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordCommandUtils.
>> materializeOneInstance(
>> > CoordCommandUtils.java:546)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>> > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
>> >
>> > at
>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>> > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
>> >
>> > at
>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>> > MaterializeTransitionXCommand.java:73)
>> >
>> > at
>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>> > MaterializeTransitionXCommand.java:29)
>> >
>> > at org.apache.oozie.command.XCommand.call(XCommand.java:290)
>> >
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >
>> > at
>> > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
>> > CallableQueueService.java:181)
>> >
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1149)
>> >
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:624)
>> >
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> >
>> >
>> > Is S3 support specific to CDH distribution or should it work in Apache
>> > Oozie as well? I’m not using CDH yet so
>> >
>> > On Wed, May 16, 2018 at 10:28 AM Peter Cseh 
>> wrote:
>> >
>> > > I think it should be possible for Oozie to poll S3. Check out this
>> > > <
>> > > https://www.cloudera.com/documentation/enterprise/5-9-
>> > x/topics/admin_oozie_s3.html
>> > > >
>> > > description on how to make it work in jobs, something similar should
>> work
>> > > on the server side as well
>> > >
>> > > On Tue, May 15, 2018 at 4:43 PM, purna pradeep <
>> purna2prad...@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Andras,
>> > > >
>> > > > Also I also would like to know if oozie supports Aws S3 as input
>> events
>> > > to
>> > > > poll for a dependency file before kicking off a spark action
>> > > >
>> > > >
>> > > > For example: I don’t want to kick off a spark action until a file is
>> > > > arrived on a given AWS s3 location
>> > > >
>> > > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
>> > andras.pi...@cloudera.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > Oozie needs HDFS to store workflow, coordinator, or bundle
>> > definitions,
>> > > > as
>> > > > > well as sharelib files in a safe, distributed and scalable way.
>> Oozie
>> > > > needs
>> > > > > YARN to run almost all of its actions, Spark action being no
>> > exception.
>> > > > >
>> > > > > At the moment it's not feasible to install Oozie without those
>> Hadoop
>> > > > > components. How to install Oozie please *find here
>> > > > > 

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

This is what I already have in my oozie-site.xml




oozie.service.HadoopAccessorService.supported.filesystems

*



On Wed, May 16, 2018 at 11:37 AM Peter Cseh  wrote:

> You'll have to configure
> oozie.service.HadoopAccessorService.supported.filesystems
> hdfs,hftp,webhdfs Enlist
> the different filesystems supported for federation. If wildcard "*" is
> specified, then ALL file schemes will be allowed.properly.
>
> For testing purposes it's ok to put * in there in oozie-site.xml
>
> On Wed, May 16, 2018 at 5:29 PM, purna pradeep 
> wrote:
>
> > Peter,
> >
> > I have tried to specify dataset with uri starting with s3://, s3a:// and
> > s3n:// and I am getting exception
> >
> >
> >
> > Exception occurred:E0904: Scheme [s3] not supported in uri
> > [s3://mybucket/input.data] Making the job failed
> >
> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
> > supported in uri [s3:// mybucket /input.data]
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:185)
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:168)
> >
> > at
> > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > URIHandlerService.java:160)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
> > CoordCommandUtils.java:465)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.
> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.
> > materializeInputDataEvents(CoordCommandUtils.java:731)
> >
> > at
> > org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
> > CoordCommandUtils.java:546)
> >
> > at
> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
> >
> > at
> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
> >
> > at
> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
> > MaterializeTransitionXCommand.java:73)
> >
> > at
> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
> > MaterializeTransitionXCommand.java:29)
> >
> > at org.apache.oozie.command.XCommand.call(XCommand.java:290)
> >
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >
> > at
> > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
> > CallableQueueService.java:181)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1149)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:624)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> >
> >
> > Is S3 support specific to CDH distribution or should it work in Apache
> > Oozie as well? I’m not using CDH yet so
> >
> > On Wed, May 16, 2018 at 10:28 AM Peter Cseh 
> wrote:
> >
> > > I think it should be possible for Oozie to poll S3. Check out this
> > > <
> > > https://www.cloudera.com/documentation/enterprise/5-9-
> > x/topics/admin_oozie_s3.html
> > > >
> > > description on how to make it work in jobs, something similar should
> work
> > > on the server side as well
> > >
> > > On Tue, May 15, 2018 at 4:43 PM, purna pradeep <
> purna2prad...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Andras,
> > > >
> > > > Also I also would like to know if oozie supports Aws S3 as input
> events
> > > to
> > > > poll for a dependency file before kicking off a spark action
> > > >
> > > >
> > > > For example: I don’t want to kick off a spark action until a file is
> > > > arrived on a given AWS s3 location
> > > >
> > > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
> > andras.pi...@cloudera.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Oozie needs HDFS to store workflow, coordinator, or bundle
> > definitions,
> > > > as
> > > > > well as sharelib files in a safe, distributed and scalable way.
> Oozie
> > > > needs
> > > > > YARN to run almost all of its actions, Spark action being no
> > exception.
> > > > >
> > > > > At the moment it's not feasible to install Oozie without those
> Hadoop
> > > > > components. How to install Oozie please *find here
> > > > > *.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Andras
> > > > >
> > > > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
> > > purna2prad...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Would like to know if I can use sparkaction in oozie without
> having
> > > > > Hadoop
> > > > > > cluster?
> > > > > >
> > > > > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > > > > >
> > > > > > I’m a beginner in oozie
> > > > > >
> > > > > > Thanks

Re: Oozie for spark jobs without Hadoop

[Peter Cseh]

You'll have to configure
oozie.service.HadoopAccessorService.supported.filesystems
hdfs,hftp,webhdfs Enlist
the different filesystems supported for federation. If wildcard "*" is
specified, then ALL file schemes will be allowed.properly.

For testing purposes it's ok to put * in there in oozie-site.xml

On Wed, May 16, 2018 at 5:29 PM, purna pradeep 
wrote:

> Peter,
>
> I have tried to specify dataset with uri starting with s3://, s3a:// and
> s3n:// and I am getting exception
>
>
>
> Exception occurred:E0904: Scheme [s3] not supported in uri
> [s3://mybucket/input.data] Making the job failed
>
> org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
> supported in uri [s3:// mybucket /input.data]
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(
> URIHandlerService.java:185)
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(
> URIHandlerService.java:168)
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(
> URIHandlerService.java:160)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
> CoordCommandUtils.java:465)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.
> separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.
> materializeInputDataEvents(CoordCommandUtils.java:731)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
> CoordCommandUtils.java:546)
>
> at
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
>
> at
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
> mand.materialize(CoordMaterializeTransitionXCommand.java:362)
>
> at
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(
> MaterializeTransitionXCommand.java:73)
>
> at
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(
> MaterializeTransitionXCommand.java:29)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:290)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
> CallableQueueService.java:181)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is S3 support specific to CDH distribution or should it work in Apache
> Oozie as well? I’m not using CDH yet so
>
> On Wed, May 16, 2018 at 10:28 AM Peter Cseh  wrote:
>
> > I think it should be possible for Oozie to poll S3. Check out this
> > <
> > https://www.cloudera.com/documentation/enterprise/5-9-
> x/topics/admin_oozie_s3.html
> > >
> > description on how to make it work in jobs, something similar should work
> > on the server side as well
> >
> > On Tue, May 15, 2018 at 4:43 PM, purna pradeep 
> > wrote:
> >
> > > Thanks Andras,
> > >
> > > Also I also would like to know if oozie supports Aws S3 as input events
> > to
> > > poll for a dependency file before kicking off a spark action
> > >
> > >
> > > For example: I don’t want to kick off a spark action until a file is
> > > arrived on a given AWS s3 location
> > >
> > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
> andras.pi...@cloudera.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Oozie needs HDFS to store workflow, coordinator, or bundle
> definitions,
> > > as
> > > > well as sharelib files in a safe, distributed and scalable way. Oozie
> > > needs
> > > > YARN to run almost all of its actions, Spark action being no
> exception.
> > > >
> > > > At the moment it's not feasible to install Oozie without those Hadoop
> > > > components. How to install Oozie please *find here
> > > > *.
> > > >
> > > > Regards,
> > > >
> > > > Andras
> > > >
> > > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
> > purna2prad...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Would like to know if I can use sparkaction in oozie without having
> > > > Hadoop
> > > > > cluster?
> > > > >
> > > > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > > > >
> > > > > I’m a beginner in oozie
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com 
> >
> > [image: Cloudera] 
> >
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> Cloudera
> > on LinkedIn] 
> > --
> >
>



-- 
*Peter Cseh *| Software Engineer
cloudera.com 

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

+Peter

On Wed, May 16, 2018 at 11:29 AM purna pradeep 
wrote:

> Peter,
>
> I have tried to specify dataset with uri starting with s3://, s3a:// and
> s3n:// and I am getting exception
>
>
>
> Exception occurred:E0904: Scheme [s3] not supported in uri
> [s3://mybucket/input.data] Making the job failed
>
> org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
> supported in uri [s3:// mybucket /input.data]
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185)
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168)
>
> at
> org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731)
>
> at
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546)
>
> at
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
>
> at
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362)
>
> at
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73)
>
> at
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:290)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is S3 support specific to CDH distribution or should it work in Apache
> Oozie as well? I’m not using CDH yet so
>
> On Wed, May 16, 2018 at 10:28 AM Peter Cseh  wrote:
>
>> I think it should be possible for Oozie to poll S3. Check out this
>> <
>> https://www.cloudera.com/documentation/enterprise/5-9-x/topics/admin_oozie_s3.html
>> >
>> description on how to make it work in jobs, something similar should work
>> on the server side as well
>>
>> On Tue, May 15, 2018 at 4:43 PM, purna pradeep 
>> wrote:
>>
>> > Thanks Andras,
>> >
>> > Also I also would like to know if oozie supports Aws S3 as input events
>> to
>> > poll for a dependency file before kicking off a spark action
>> >
>> >
>> > For example: I don’t want to kick off a spark action until a file is
>> > arrived on a given AWS s3 location
>> >
>> > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
>> andras.pi...@cloudera.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Oozie needs HDFS to store workflow, coordinator, or bundle
>> definitions,
>> > as
>> > > well as sharelib files in a safe, distributed and scalable way. Oozie
>> > needs
>> > > YARN to run almost all of its actions, Spark action being no
>> exception.
>> > >
>> > > At the moment it's not feasible to install Oozie without those Hadoop
>> > > components. How to install Oozie please *find here
>> > > *.
>> > >
>> > > Regards,
>> > >
>> > > Andras
>> > >
>> > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
>> purna2prad...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > Would like to know if I can use sparkaction in oozie without having
>> > > Hadoop
>> > > > cluster?
>> > > >
>> > > > I want to use oozie to schedule spark jobs on Kubernetes cluster
>> > > >
>> > > > I’m a beginner in oozie
>> > > >
>> > > > Thanks
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> *Peter Cseh *| Software Engineer
>> cloudera.com 
>>
>> [image: Cloudera] 
>>
>> [image: Cloudera on Twitter]  [image:
>> Cloudera on Facebook]  [image:
>> Cloudera
>> on LinkedIn] 
>> --
>>
>

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

Peter,

I have tried to specify dataset with uri starting with s3://, s3a:// and
s3n:// and I am getting exception



Exception occurred:E0904: Scheme [s3] not supported in uri
[s3://mybucket/input.data] Making the job failed

org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
supported in uri [s3:// mybucket /input.data]

at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185)

at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168)

at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160)

at
org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465)

at
org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404)

at
org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731)

at
org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546)

at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492)

at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362)

at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73)

at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29)

at org.apache.oozie.command.XCommand.call(XCommand.java:290)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)



Is S3 support specific to CDH distribution or should it work in Apache
Oozie as well? I’m not using CDH yet so

On Wed, May 16, 2018 at 10:28 AM Peter Cseh  wrote:

> I think it should be possible for Oozie to poll S3. Check out this
> <
> https://www.cloudera.com/documentation/enterprise/5-9-x/topics/admin_oozie_s3.html
> >
> description on how to make it work in jobs, something similar should work
> on the server side as well
>
> On Tue, May 15, 2018 at 4:43 PM, purna pradeep 
> wrote:
>
> > Thanks Andras,
> >
> > Also I also would like to know if oozie supports Aws S3 as input events
> to
> > poll for a dependency file before kicking off a spark action
> >
> >
> > For example: I don’t want to kick off a spark action until a file is
> > arrived on a given AWS s3 location
> >
> > On Tue, May 15, 2018 at 10:17 AM Andras Piros  >
> > wrote:
> >
> > > Hi,
> > >
> > > Oozie needs HDFS to store workflow, coordinator, or bundle definitions,
> > as
> > > well as sharelib files in a safe, distributed and scalable way. Oozie
> > needs
> > > YARN to run almost all of its actions, Spark action being no exception.
> > >
> > > At the moment it's not feasible to install Oozie without those Hadoop
> > > components. How to install Oozie please *find here
> > > *.
> > >
> > > Regards,
> > >
> > > Andras
> > >
> > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
> purna2prad...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Would like to know if I can use sparkaction in oozie without having
> > > Hadoop
> > > > cluster?
> > > >
> > > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > > >
> > > > I’m a beginner in oozie
> > > >
> > > > Thanks
> > > >
> > >
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com 
>
> [image: Cloudera] 
>
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
>

Re: Oozie for spark jobs without Hadoop

[Shikin, Igor]

Hi Peter,
I am working with Purna. I have tried to specify dataset with uri starting with 
s3://, s3a:// and s3n:// and I am getting exception

Exception occurred:E0904: Scheme [s3] not supported in uri 
[s3://cmsegmentation-qa/oozie-test/input.data] Making the job failed 
org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not 
supported in uri [s3://cmsegmentation-qa/oozie-test/input.data]
at 
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185)
at 
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168)
at 
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160)
at 
org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465)
at 
org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404)
at 
org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731)
at 
org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546)
at 
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
at 
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362)
at 
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73)
at 
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Is S3 support specific to CDH distribution or should in work in Apache Oozie as 
well?

Thanks!

On 5/16/18, 10:29 AM, "Peter Cseh"  wrote:

I think it should be possible for Oozie to poll S3. Check out this


description on how to make it work in jobs, something similar should work
on the server side as well

On Tue, May 15, 2018 at 4:43 PM, purna pradeep 
wrote:

> Thanks Andras,
>
> Also I also would like to know if oozie supports Aws S3 as input events to
> poll for a dependency file before kicking off a spark action
>
>
> For example: I don’t want to kick off a spark action until a file is
> arrived on a given AWS s3 location
>
> On Tue, May 15, 2018 at 10:17 AM Andras Piros 
> wrote:
>
> > Hi,
> >
> > Oozie needs HDFS to store workflow, coordinator, or bundle definitions,
> as
> > well as sharelib files in a safe, distributed and scalable way. Oozie
> needs
> > YARN to run almost all of its actions, Spark action being no exception.
> >
> > At the moment it's not feasible to install Oozie without those Hadoop
> > components. How to install Oozie please *find here
> > 
*.
> >
> > Regards,
> >
> > Andras
> >
> > On Tue, May 15, 2018 at 4:11 PM, purna pradeep 
> > wrote:
> >
> > > Hi,
> > >
> > > Would like to know if I can use sparkaction in oozie without having
> > Hadoop
> > > cluster?
> > >
> > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > >
> > > I’m a beginner in oozie
> > >
> > > Thanks
> > >
> >
>



-- 
*Peter Cseh *| Software Engineer
cloudera.com 


[image: Cloudera] 

Re: Oozie for spark jobs without Hadoop

[Peter Cseh]

I think it should be possible for Oozie to poll S3. Check out this

description on how to make it work in jobs, something similar should work
on the server side as well

On Tue, May 15, 2018 at 4:43 PM, purna pradeep 
wrote:

> Thanks Andras,
>
> Also I also would like to know if oozie supports Aws S3 as input events to
> poll for a dependency file before kicking off a spark action
>
>
> For example: I don’t want to kick off a spark action until a file is
> arrived on a given AWS s3 location
>
> On Tue, May 15, 2018 at 10:17 AM Andras Piros 
> wrote:
>
> > Hi,
> >
> > Oozie needs HDFS to store workflow, coordinator, or bundle definitions,
> as
> > well as sharelib files in a safe, distributed and scalable way. Oozie
> needs
> > YARN to run almost all of its actions, Spark action being no exception.
> >
> > At the moment it's not feasible to install Oozie without those Hadoop
> > components. How to install Oozie please *find here
> > *.
> >
> > Regards,
> >
> > Andras
> >
> > On Tue, May 15, 2018 at 4:11 PM, purna pradeep 
> > wrote:
> >
> > > Hi,
> > >
> > > Would like to know if I can use sparkaction in oozie without having
> > Hadoop
> > > cluster?
> > >
> > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > >
> > > I’m a beginner in oozie
> > >
> > > Thanks
> > >
> >
>



-- 
*Peter Cseh *| Software Engineer
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

Re: Spark 2.3 in oozie

[Peter Cseh]

The version of the xml schema has nothing to do with the version of the
component you're using.

Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
for compilation with Spark 2.3.0

Oozie does not pull in Spark's Kubernetes artifact.
To make it part of the Oozie Spark sharelib you'll have to include the
spark-kubernetes.jar

in
the sharelib/spark/pom.xml as a compile-time dependency.

gp

On Tue, May 15, 2018 at 9:04 PM, purna pradeep 
wrote:

> I’m able to compile successfully with after adding this override option
>
> -Dspark.scala.binary.version=2.11
>
> Dspark.version = 2.3.0
>
> But when I’m running a spark action with spark-pi example jar against
> Kubernetes master I’m getting below error in stderr log
>
>
> *Error:Could not load KUBERNETES classes.This copy of spark may not have
> been compiled with Kubernetes support*
>
> Below is my workflow.xml
>
> <*spark xmlns="uri:oozie:spark-action:1.0">*
>
> *${resourceManager}*
>
> *${nameNode}*
>
> *k8s://<***.com>*
>
> *Python-Spark-Pi*
>
> *spark-examples_2.11-2.3.0.jar*
>
> *--class org.apache.spark.examples.SparkPi --conf
> spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> spark.kubernetes.container.image=artifactory.cloud.
> capitalone.com/kubespark/spark-quantum:v2.3.0
> 
> --conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
>  >
> --conf
> spark.kubernetes.driver.label.application=is1-driver --conf
> spark.kubernetes.executor.label.application=is1-exec*utor
> local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> 0.jar
>
> 
>
>
> Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> needs to be spark-action:2.0 as I’m using spark 2.3?
>
>
> Please suggest!
>
>
> On Tue, May 15, 2018 at 12:43 PM Peter Cseh  wrote:
>
> > I think the error is related to the Scala version being present in the
> > artifact name.
> > I'll take a look at this tomorrow.
> > Gp
> >
> > On Tue, May 15, 2018, 18:28 Artem Ervits  wrote:
> >
> > > Did you run
> > > mvn clean install first on the parent directory?
> > >
> > > On Tue, May 15, 2018, 11:35 AM purna pradeep 
> > > wrote:
> > >
> > > > Thanks peter,
> > > >
> > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > > > getting below error from oozie examples
> > > >
> > > >
> > > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > > resolve
> > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> > Could
> > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > resolution
> > > > *
> > > >
> > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh 
> > > wrote:
> > > >
> > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> > 2.1:
> > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> that
> > > way
> > > > it
> > > > > will work.
> > > > > gp
> > > > >
> > > > >
> > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > purna2prad...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > > > version
> > > > > >
> > > > > > I want to use spark action
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Purna
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Peter Cseh *| Software Engineer
> > > > > cloudera.com 
> > > > >
> > > > > [image: Cloudera] 
> > > > >
> > > > > [image: Cloudera on Twitter] 
> [image:
> > > > > Cloudera on Facebook]  [image:
> > > > Cloudera
> > > > > on LinkedIn] 
> > > > > --
> > > > >
> > > >
> > >
> >
>



-- 
*Peter Cseh *| Software Engineer
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--