Peter,
I have tried to specify dataset with uri starting with s3://, s3a:// and
s3n:// and I am getting exception
Exception occurred:E0904: Scheme [s3] not supported in uri
[s3://mybucket/input.data] Making the job failed
org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
supported in uri [s3:// mybucket /input.data]
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185)
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168)
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160)
at
org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465)
at
org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404)
at
org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731)
at
org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546)
at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362)
at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73)
at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is S3 support specific to CDH distribution or should it work in Apache
Oozie as well? I’m not using CDH yet so
On Wed, May 16, 2018 at 10:28 AM Peter Cseh <gezap...@cloudera.com> wrote:
> I think it should be possible for Oozie to poll S3. Check out this
> <
> https://www.cloudera.com/documentation/enterprise/5-9-x/topics/admin_oozie_s3.html
> >
> description on how to make it work in jobs, something similar should work
> on the server side as well
>
> On Tue, May 15, 2018 at 4:43 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
> > Thanks Andras,
> >
> > Also I also would like to know if oozie supports Aws S3 as input events
> to
> > poll for a dependency file before kicking off a spark action
> >
> >
> > For example: I don’t want to kick off a spark action until a file is
> > arrived on a given AWS s3 location
> >
> > On Tue, May 15, 2018 at 10:17 AM Andras Piros <andras.pi...@cloudera.com
> >
> > wrote:
> >
> > > Hi,
> > >
> > > Oozie needs HDFS to store workflow, coordinator, or bundle definitions,
> > as
> > > well as sharelib files in a safe, distributed and scalable way. Oozie
> > needs
> > > YARN to run almost all of its actions, Spark action being no exception.
> > >
> > > At the moment it's not feasible to install Oozie without those Hadoop
> > > components. How to install Oozie please *find here
> > > <https://oozie.apache.org/docs/5.0.0/AG_Install.html>*.
> > >
> > > Regards,
> > >
> > > Andras
> > >
> > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <
> purna2prad...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Would like to know if I can use sparkaction in oozie without having
> > > Hadoop
> > > > cluster?
> > > >
> > > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > > >
> > > > I’m a beginner in oozie
> > > >
> > > > Thanks
> > > >
> > >
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
