I have tried with coordinator's configuration too but no luck ☹️
On Wed, May 16, 2018 at 3:54 PM Peter Cseh <gezap...@cloudera.com> wrote:
> Great progress there purna! :)
>
> Have you tried adding these properites to the coordinator's configuration?
> we usually use the action config to build up connection to the distributed
> file system.
> Although I'm not sure we're using these when polling the dependencies for
> coordinators, but I'm excited about you trying to make it work!
>
> I'll get back with a - hopefully - more helpful answer soon, I have to
> check the code in more depth first.
> gp
>
> On Wed, May 16, 2018 at 9:45 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
>> Peter,
>>
>> I got rid of this error by adding
>> hadoop-aws-2.8.3.jar and jets3t-0.9.4.jar
>>
>> But I’m getting below error now
>>
>> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
>> Key must be specified by setting the fs.s3.awsAccessKeyId and
>> fs.s3.awsSecretAccessKey properties (respectively)
>>
>> I have tried adding AWS access ,secret keys in
>>
>> oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml
>>
>>
>>
>>
>> On Wed, May 16, 2018 at 2:30 PM purna pradeep <purna2prad...@gmail.com>
>> wrote:
>>
>>>
>>> I have tried this ,just added s3 instead of *
>>>
>>> <property>
>>>
>>>
>>> <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
>>>
>>> <value>hdfs,hftp,webhdfs,s3</value>
>>>
>>> </property>
>>>
>>>
>>> Getting below error
>>>
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.fs.s3a.S3AFileSystem not found
>>>
>>> at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)
>>>
>>> at
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)
>>>
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
>>>
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
>>>
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
>>>
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
>>>
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
>>>
>>> at
>>> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
>>>
>>> at
>>> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623
>>>
>>>
>>> On Wed, May 16, 2018 at 2:19 PM purna pradeep <purna2prad...@gmail.com>
>>> wrote:
>>>
>>>> This is what is in the logs
>>>>
>>>> 2018-05-16 14:06:13,500 INFO URIHandlerService:520 - SERVER[localhost]
>>>> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
>>>>
>>>> 2018-05-16 14:06:13,501 INFO URIHandlerService:520 - SERVER[localhost]
>>>> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
>>>>
>>>>
>>>> On Wed, May 16, 2018 at 12:27 PM Peter Cseh <gezap...@cloudera.com>
>>>> wrote:
>>>>
>>>>> That's strange, this exception should not happen in that case.
>>>>> Can you check the server logs for messages like this?
>>>>> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
>>>>> LOG.info("Loaded default urihandler {0}",
>>>>> defaultHandler.getClass().getName());
>>>>> Thanks
>>>>>
>>>>> On Wed, May 16, 2018 at 5:47 PM, purna pradeep <
>>>>> purna2prad...@gmail.com> wrote:
>>>>>
>>>>>> This is what I already have in my oozie-site.xml
>>>>>>
>>>>>> <property>
>>>>>>
>>>>>>
>>>>>> <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
>>>>>>
>>>>>> <value>*</value>
>>>>>>
>>>>>> </property>
>>>>>>
>>>>>> On Wed, May 16, 2018 at 11:37 AM Peter Cseh <gezap...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> You'll have to configure
>>>>>>> oozie.service.HadoopAccessorService.supported.filesystems
>>>>>>> hdfs,hftp,webhdfs Enlist
>>>>>>> the different filesystems supported for federation. If wildcard "*"
>>>>>>> is
>>>>>>> specified, then ALL file schemes will be allowed.properly.
>>>>>>>
>>>>>>> For testing purposes it's ok to put * in there in oozie-site.xml
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 5:29 PM, purna pradeep <
>>>>>>> purna2prad...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Peter,
>>>>>>> >
>>>>>>> > I have tried to specify dataset with uri starting with s3://,
>>>>>>> s3a:// and
>>>>>>> > s3n:// and I am getting exception
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Exception occurred:E0904: Scheme [s3] not supported in uri
>>>>>>> > [s3://mybucket/input.data] Making the job failed
>>>>>>> >
>>>>>>> > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme
>>>>>>> [s3] not
>>>>>>> > supported in uri [s3:// mybucket /input.data]
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>>>>>> > URIHandlerService.java:185)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>>>>>> > URIHandlerService.java:168)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.service.URIHandlerService.getURIHandler(
>>>>>>> > URIHandlerService.java:160)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
>>>>>>> > CoordCommandUtils.java:465)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.coord.CoordCommandUtils.
>>>>>>> > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.coord.CoordCommandUtils.
>>>>>>> > materializeInputDataEvents(CoordCommandUtils.java:731)
>>>>>>> >
>>>>>>> > at
>>>>>>> >
>>>>>>> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(
>>>>>>> > CoordCommandUtils.java:546)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>>>>>>> >
>>>>>>> mand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom
>>>>>>> > mand.materialize(CoordMaterializeTransitionXCommand.java:362)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>>>>>>> > MaterializeTransitionXCommand.java:73)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.command.MaterializeTransitionXCommand.execute(
>>>>>>> > MaterializeTransitionXCommand.java:29)
>>>>>>> >
>>>>>>> > at org.apache.oozie.command.XCommand.call(XCommand.java:290)
>>>>>>> >
>>>>>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>> >
>>>>>>> > at
>>>>>>> > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(
>>>>>>> > CallableQueueService.java:181)
>>>>>>> >
>>>>>>> > at
>>>>>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> > ThreadPoolExecutor.java:1149)
>>>>>>> >
>>>>>>> > at
>>>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> > ThreadPoolExecutor.java:624)
>>>>>>> >
>>>>>>> > at java.lang.Thread.run(Thread.java:748)
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Is S3 support specific to CDH distribution or should it work in
>>>>>>> Apache
>>>>>>> > Oozie as well? I’m not using CDH yet so
>>>>>>> >
>>>>>>> > On Wed, May 16, 2018 at 10:28 AM Peter Cseh <gezap...@cloudera.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > I think it should be possible for Oozie to poll S3. Check out
>>>>>>> this
>>>>>>> > > <
>>>>>>> > > https://www.cloudera.com/documentation/enterprise/5-9-
>>>>>>> > x/topics/admin_oozie_s3.html
>>>>>>> > > >
>>>>>>> > > description on how to make it work in jobs, something similar
>>>>>>> should work
>>>>>>> > > on the server side as well
>>>>>>> > >
>>>>>>> > > On Tue, May 15, 2018 at 4:43 PM, purna pradeep <
>>>>>>> purna2prad...@gmail.com>
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > > > Thanks Andras,
>>>>>>> > > >
>>>>>>> > > > Also I also would like to know if oozie supports Aws S3 as
>>>>>>> input events
>>>>>>> > > to
>>>>>>> > > > poll for a dependency file before kicking off a spark action
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > For example: I don’t want to kick off a spark action until a
>>>>>>> file is
>>>>>>> > > > arrived on a given AWS s3 location
>>>>>>> > > >
>>>>>>> > > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <
>>>>>>> > andras.pi...@cloudera.com
>>>>>>> > > >
>>>>>>> > > > wrote:
>>>>>>> > > >
>>>>>>> > > > > Hi,
>>>>>>> > > > >
>>>>>>> > > > > Oozie needs HDFS to store workflow, coordinator, or bundle
>>>>>>> > definitions,
>>>>>>> > > > as
>>>>>>> > > > > well as sharelib files in a safe, distributed and scalable
>>>>>>> way. Oozie
>>>>>>> > > > needs
>>>>>>> > > > > YARN to run almost all of its actions, Spark action being no
>>>>>>> > exception.
>>>>>>> > > > >
>>>>>>> > > > > At the moment it's not feasible to install Oozie without
>>>>>>> those Hadoop
>>>>>>> > > > > components. How to install Oozie please *find here
>>>>>>> > > > > <https://oozie.apache.org/docs/5.0.0/AG_Install.html>*.
>>>>>>> > > > >
>>>>>>> > > > > Regards,
>>>>>>> > > > >
>>>>>>> > > > > Andras
>>>>>>> > > > >
>>>>>>> > > > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
>>>>>>> > > purna2prad...@gmail.com>
>>>>>>> > > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > > Hi,
>>>>>>> > > > > >
>>>>>>> > > > > > Would like to know if I can use sparkaction in oozie
>>>>>>> without having
>>>>>>> > > > > Hadoop
>>>>>>> > > > > > cluster?
>>>>>>> > > > > >
>>>>>>> > > > > > I want to use oozie to schedule spark jobs on Kubernetes
>>>>>>> cluster
>>>>>>> > > > > >
>>>>>>> > > > > > I’m a beginner in oozie
>>>>>>> > > > > >
>>>>>>> > > > > > Thanks
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> > > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > --
>>>>>>> > > *Peter Cseh *| Software Engineer
>>>>>>> > > cloudera.com <https://www.cloudera.com>
>>>>>>> > >
>>>>>>> > > [image: Cloudera] <https://www.cloudera.com/>
>>>>>>> > >
>>>>>>> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>>>>>>> [image:
>>>>>>> > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>>>>>>> [image:
>>>>>>> > Cloudera
>>>>>>> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>>>>> > > ------------------------------
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Peter Cseh *| Software Engineer
>>>>>>> cloudera.com <https://www.cloudera.com>
>>>>>>>
>>>>>>> [image: Cloudera] <https://www.cloudera.com/>
>>>>>>>
>>>>>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>>>>>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>>>>>> Cloudera
>>>>>>> on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Peter Cseh *| Software Engineer
>>>>> cloudera.com <https://www.cloudera.com>
>>>>>
>>>>> [image: Cloudera] <https://www.cloudera.com/>
>>>>>
>>>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>>>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>>>>> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
>>>>> ------------------------------
>>>>>
>>>>>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
>
