There is no need for this setting if you have the hadoop configs under conf/hadoop-conf. Having this setting would be more useful if you have oozie configured to talk with multiple hadoops. This configs are now required at startup as the share lib service needs to connect to the filesystem on which the share lib jars need to be copied (probably only require the core-site.xml, need to check) Before also it was recommended to have the hadoop configs on oozie-server, so users don't need to define some of this configurations in their workflows. As now its mandatory, we should make it clear in our documentation.
Thanks, Virag From: bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> Reply-To: bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> Date: Thursday, September 26, 2013 12:20 PM To: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" <dev@oozie.apache.org<mailto:dev@oozie.apache.org>>, Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> Subject: Re: issue after OOZIE-1461 I second on Robert's concern. Right now, even the easiest way for me to get around this is to point " oozie.service.HadoopAccessorService.hadoop.configurations" to "*=Absolute path of my hadoop conf". Bowen ________________________________ From: Robert Kanter <rkan...@cloudera.com<mailto:rkan...@cloudera.com>> To: Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> Cc: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" <dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> Sent: Thursday, September 26, 2013 11:11 AM Subject: Re: issue after OOZIE-1461 Is there any way to make that step not required, or to at least streamline it somehow? I imagine we'll see many questions from users wondering why their Oozie server doesn't start because of this. Also, Oozie used to work out-of-the-box after running a few scripts; now it requires manually setting the oozie.service.HadoopAccessorService.hadoop.configurations property. thanks - Robert On Mon, Sep 23, 2013 at 2:57 PM, Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> wrote: > Bowen, > > Including hadoop configs on oozie-server is documented at > https://oozie.apache.org/docs/3.3.2/AG_HadoopConfiguration.html. > Documentation for sharelib installation can be updated once OOZIE-1518 and > OOZIE-1519 goes in. We can also update quick start guide during that time. > > Regards, > Virag > > > On 9/23/13 2:31 PM, "bowen zhang" > <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> wrote: > > >Virag, > >Can you add documentation for this change since the current build and > >setup of oozie doesn't cover this? > >Bowen > > > > > > > > > >________________________________ > > From: Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> > >To: bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>; > >"dev@oozie.apache.org<mailto:dev@oozie.apache.org>" > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; Robert Kanter > ><rkan...@cloudera.com<mailto:rkan...@cloudera.com>> > >Sent: Monday, September 16, 2013 2:58 PM > >Subject: Re: issue after OOZIE-1461 > > > > > >Hi Robert/Bowen, > > > >The hadoop configs need to be there in class path (hadoop-conf dir or > >oozie-server/lib). So the HadoopAccessorService can create the > >appropriate filesystem object. This will fix your current issue. > >But there is one more problem you might face while running a job where > >permissions are not recursively applied. This fix is in OOZIE-1528 and > >will be checked in shortly. > > > >Thanks, > >Virag > > > >From: bowen zhang > ><bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com><mailto:bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> > >Reply-To: bowen zhang > ><bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com><mailto:bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> > >Date: Monday, September 16, 2013 2:49 PM > >To: > >"dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto:dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto:dev@oozie.apache.org<mailto:dev@oozie.apache.org>>>, > > Virag Kothari > ><vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com><mailto:vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>> > >Subject: Re: Fwd: issue after OOZIE-1461 > > > >what I found is the variable uri from tmpShareLibPath has authority of > >"null". > > > > > >________________________________ > >From: Robert Kanter > ><rkan...@cloudera.com<mailto:rkan...@cloudera.com><mailto:rkan...@cloudera.com<mailto:rkan...@cloudera.com>>> > >To: Virag Kothari > ><vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com><mailto:vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>>; > >"dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto:dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto:dev@oozie.apache.org<mailto:dev@oozie.apache.org>>> > >Sent: Monday, September 16, 2013 2:39 PM > >Subject: Fwd: issue after OOZIE-1461 > > > >Hi Virag, > > > >After OOZIE-1461, Bowen (and I too) have run into this exception when > >starting Oozie, so it fails and won't start. I checked, and for me at > >least, the share/lib/ dir looks like it has the correct permissions. > >Any thoughts? > > > >thanks > >- Robert > > > > > >On Mon, Sep 16, 2013 at 2:26 PM, Bowen Zhang > ><bzh...@hortonworks.com<mailto:bzh...@hortonworks.com><mailto:bzh...@hortonworks.com<mailto:bzh...@hortonworks.com>>> > > wrote: > > > >> Hi Robert, > >> After rebasing to the trunk, I hit this error when trying to bring up > >> oozie. > >> org.apache.oozie.service.ServiceException: E0100: Could not initialize > >> service [org.apache.oozie.service.ShareLibService], Failed to set > >> permissions of path: /user/bzhang/share/lib/tmp-20130916135406/oozie to > >>0755 > >> at > >> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:81) > >> at > >> org.apache.oozie.service.Services.setServiceInternal(Services.java:368) > >> at > >>org.apache.oozie.service.Services.setService(Services.java:354) > >> at > >> org.apache.oozie.service.Services.loadServices(Services.java:287) > >> at org.apache.oozie.service.Services.init(Services.java:208) > >> at > >> > >>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader > >>.java:45) > >> at > >> > >>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.ja > >>va:4206) > >> at > >> > >>org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) > >> at > >> > >>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.jav > >>a:799) > >> at > >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) > >> at > >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) > >> at > >> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) > >> at > >> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) > >> at > >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) > >> at > >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) > >> at > >> > >>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324 > >>) > >> at > >> > >>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSup > >>port.java:142) > >> at > >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) > >> at > >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840) > >> at > >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) > >> at > >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) > >> at > >> org.apache.catalina.core.StandardService.start(StandardService.java:525) > >> at > >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754) > >> at org.apache.catalina.startup.Catalina.start(Catalina.java:595) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java > >>:39) > >> at > >> > >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > >>mpl.java:25) > >> at java.lang.reflect.Method.invoke(Method.java:597) > >> at > >>org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > >> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) > >> Caused by: java.io.IOException: Failed to set permissions of path: > >> /user/bzhang/share/lib/tmp-20130916135406/oozie to 0755 > >> at > >> org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) > >> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662) > >> at > >> > >>org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem. > >>java:509) > >> at > >> > >>org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java > >>:286) > >> at > >> > >>org.apache.oozie.service.ShareLibService.copyJarContainingClasses(ShareLi > >>bService.java:109) > >> at > >> > >>org.apache.oozie.service.ShareLibService.copyLauncherJarsToShareLib(Share > >>LibService.java:90) > >> at > >> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:76) > >> ... 29 more > >> > >> > >> It might have sth to do with the version of hadoop that hadoopAccessor > >> cannot change file permission. Do you see this problem? > >> > >> > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or entity > >> to which it is addressed and may contain information that is > >>confidential, > >> privileged and exempt from disclosure under applicable law. If the > >>reader > >> of this message is not the intended recipient, you are hereby notified > >>that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > >>immediately > >> and delete it from your system. Thank You. > >
