Hi Peter, I am working with Purna. I have tried to specify dataset with uri starting with s3://, s3a:// and s3n:// and I am getting exception
Exception occurred:E0904: Scheme [s3] not supported in uri [s3://cmsegmentation-qa/oozie-test/input.data] Making the job failed org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not supported in uri [s3://cmsegmentation-qa/oozie-test/input.data] at org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185) at org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168) at org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160) at org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465) at org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404) at org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731) at org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546) at org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492) at org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362) at org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73) at org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29) at org.apache.oozie.command.XCommand.call(XCommand.java:290) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Is S3 support specific to CDH distribution or should in work in Apache Oozie as well? Thanks! On 5/16/18, 10:29 AM, "Peter Cseh" <gezap...@cloudera.com> wrote: I think it should be possible for Oozie to poll S3. Check out this <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com_documentation_enterprise_5-2D9-2Dx_topics_admin-5Foozie-5Fs3.html&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=dGiatnV6rgwui2UT5_IPwJR0CLUiRosukYp1v9Wa6ig&e=> description on how to make it work in jobs, something similar should work on the server side as well On Tue, May 15, 2018 at 4:43 PM, purna pradeep <purna2prad...@gmail.com> wrote: > Thanks Andras, > > Also I also would like to know if oozie supports Aws S3 as input events to > poll for a dependency file before kicking off a spark action > > > For example: I don’t want to kick off a spark action until a file is > arrived on a given AWS s3 location > > On Tue, May 15, 2018 at 10:17 AM Andras Piros <andras.pi...@cloudera.com> > wrote: > > > Hi, > > > > Oozie needs HDFS to store workflow, coordinator, or bundle definitions, > as > > well as sharelib files in a safe, distributed and scalable way. Oozie > needs > > YARN to run almost all of its actions, Spark action being no exception. > > > > At the moment it's not feasible to install Oozie without those Hadoop > > components. How to install Oozie please *find here > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__oozie.apache.org_docs_5.0.0_AG-5FInstall.html&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=YRCcmhnMovHP2GiPKZ59gzGrZjKvw_rfPJ_hqeOu_qY&e=>*. > > > > Regards, > > > > Andras > > > > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <purna2prad...@gmail.com> > > wrote: > > > > > Hi, > > > > > > Would like to know if I can use sparkaction in oozie without having > > Hadoop > > > cluster? > > > > > > I want to use oozie to schedule spark jobs on Kubernetes cluster > > > > > > I’m a beginner in oozie > > > > > > Thanks > > > > > > -- *Peter Cseh *| Software Engineer cloudera.com <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=jHSKCL-Qn2Rqhd4R10JbyR84_tOemXIG7k7DXYFx_nY&e=> [image: Cloudera] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com_&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=yyz_zjLclExig_qrlg5ACyK8h0FuBcO1ueVlTbj0-QY&e=> [image: Cloudera on Twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=dBGBZEcTe7Dk0WoYS4uXo2iL-EcyctZdhD5JLqKH0NM&e=> [image: Cloudera on Facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=W1Prrd87Wvqmug5DhHrb4yeaqZ_sd-HWeyDQonNf3b8&e=> [image: Cloudera on LinkedIn] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=NDwrLruguxGxGEV1u-MqV5_PzYWh6Uet16GkSZIvl1M&e=> ------------------------------ ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.