This is what I already have in my oozie-site.xml <property>
<name>oozie.service.HadoopAccessorService.supported.filesystems</name> <value>*</value> </property> On Wed, May 16, 2018 at 11:37 AM Peter Cseh <gezap...@cloudera.com> wrote: > You'll have to configure > oozie.service.HadoopAccessorService.supported.filesystems > hdfs,hftp,webhdfs Enlist > the different filesystems supported for federation. If wildcard "*" is > specified, then ALL file schemes will be allowed.properly. > > For testing purposes it's ok to put * in there in oozie-site.xml > > On Wed, May 16, 2018 at 5:29 PM, purna pradeep <purna2prad...@gmail.com> > wrote: > > > Peter, > > > > I have tried to specify dataset with uri starting with s3://, s3a:// and > > s3n:// and I am getting exception > > > > > > > > Exception occurred:E0904: Scheme [s3] not supported in uri > > [s3://mybucket/input.data] Making the job failed > > > > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not > > supported in uri [s3:// mybucket /input.data] > > > > at > > org.apache.oozie.service.URIHandlerService.getURIHandler( > > URIHandlerService.java:185) > > > > at > > org.apache.oozie.service.URIHandlerService.getURIHandler( > > URIHandlerService.java:168) > > > > at > > org.apache.oozie.service.URIHandlerService.getURIHandler( > > URIHandlerService.java:160) > > > > at > > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs( > > CoordCommandUtils.java:465) > > > > at > > org.apache.oozie.command.coord.CoordCommandUtils. > > separateResolvedAndUnresolved(CoordCommandUtils.java:404) > > > > at > > org.apache.oozie.command.coord.CoordCommandUtils. > > materializeInputDataEvents(CoordCommandUtils.java:731) > > > > at > > org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance( > > CoordCommandUtils.java:546) > > > > at > > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom > > mand.materializeActions(CoordMaterializeTransitionXCommand.java:492) > > > > at > > org.apache.oozie.command.coord.CoordMaterializeTransitionXCom > > mand.materialize(CoordMaterializeTransitionXCommand.java:362) > > > > at > > org.apache.oozie.command.MaterializeTransitionXCommand.execute( > > MaterializeTransitionXCommand.java:73) > > > > at > > org.apache.oozie.command.MaterializeTransitionXCommand.execute( > > MaterializeTransitionXCommand.java:29) > > > > at org.apache.oozie.command.XCommand.call(XCommand.java:290) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at > > org.apache.oozie.service.CallableQueueService$CallableWrapper.run( > > CallableQueueService.java:181) > > > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1149) > > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:624) > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > Is S3 support specific to CDH distribution or should it work in Apache > > Oozie as well? I’m not using CDH yet so > > > > On Wed, May 16, 2018 at 10:28 AM Peter Cseh <gezap...@cloudera.com> > wrote: > > > > > I think it should be possible for Oozie to poll S3. Check out this > > > < > > > https://www.cloudera.com/documentation/enterprise/5-9- > > x/topics/admin_oozie_s3.html > > > > > > > description on how to make it work in jobs, something similar should > work > > > on the server side as well > > > > > > On Tue, May 15, 2018 at 4:43 PM, purna pradeep < > purna2prad...@gmail.com> > > > wrote: > > > > > > > Thanks Andras, > > > > > > > > Also I also would like to know if oozie supports Aws S3 as input > events > > > to > > > > poll for a dependency file before kicking off a spark action > > > > > > > > > > > > For example: I don’t want to kick off a spark action until a file is > > > > arrived on a given AWS s3 location > > > > > > > > On Tue, May 15, 2018 at 10:17 AM Andras Piros < > > andras.pi...@cloudera.com > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Oozie needs HDFS to store workflow, coordinator, or bundle > > definitions, > > > > as > > > > > well as sharelib files in a safe, distributed and scalable way. > Oozie > > > > needs > > > > > YARN to run almost all of its actions, Spark action being no > > exception. > > > > > > > > > > At the moment it's not feasible to install Oozie without those > Hadoop > > > > > components. How to install Oozie please *find here > > > > > <https://oozie.apache.org/docs/5.0.0/AG_Install.html>*. > > > > > > > > > > Regards, > > > > > > > > > > Andras > > > > > > > > > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep < > > > purna2prad...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Would like to know if I can use sparkaction in oozie without > having > > > > > Hadoop > > > > > > cluster? > > > > > > > > > > > > I want to use oozie to schedule spark jobs on Kubernetes cluster > > > > > > > > > > > > I’m a beginner in oozie > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > *Peter Cseh *| Software Engineer > > > cloudera.com <https://www.cloudera.com> > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > Cloudera > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > > > > > > > -- > *Peter Cseh *| Software Engineer > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ >