Sorry, I realize that I wasn't clear on that point. The Main classes would go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but the other few classes that all actions use in the launcher jar (e.g. LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which IIRC is already included when you use sharelibs (for example, when you use the sharelib with the Pig action, it includes both the share/lib/pig and share/lib/oozie).
thanks - Robert On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <[email protected]> wrote: > Hi Robert, > > I didn't understand completely. Are you suggesting to have all action > specific main classes in /share/lib/oozie instead of their respective > action sharelibs? > > Thanks, > Virag > > From: Robert Kanter <[email protected]> > Date: Monday, May 6, 2013 12:47 PM > To: "[email protected]" <[email protected]>, Virag Kothari < > [email protected]>, Alejandro Abdelnur <[email protected]> > Subject: Re: OOZIE-1311 > > Virag, > I talked more with Alejandro about the sharelib and the launcher jar. > > It would be good if we could eventually get rid of the launcher jar > completely because: > - less files would be copied to HDFS so it should be faster > - it would fix an issue where some linux distress cleanup the temp dir and > delete the launcher jar > > The contents of the launcher jar could be moved to the Oozie sharelib > (/share/lib/oozie) which IIRC currently only has a json jar file. > > As for backwards compatibility, we could do something like what you > suggested for the other sharelibs where the webapp module includes them. > Then we'd add some property to oozie-site that would switch between the > old behavior with the launcher jar (sharelib is not required) and the new > behavior without the launcher jar (sharelib is required). A new major > release, like Oozie 4, would be a good time to introduce this change as the > default; users can use the property to go back to the old behavior. Also, > if set to use the new behavior, we should make oozie.use.system.libpath in > the job.properties default to true because the sharelib would be required, > and if set to the old behavior, it would default to false. > > Thoughts? > > thanks > - Robert > > > > > > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <[email protected]>wrote: > >> Thanks Robert. Created OOZIE-1341 >> >> On 4/24/13 11:36 AM, "Robert Kanter" <[email protected]> wrote: >> >> >I think I follow what you're saying; assuming everything works correctly, >> >I >> >think that sounds good. Can you make a JIRA with that and put up a >> patch; >> >that might be easier to see how this would work. >> > >> >thanks >> >- Robert >> > >> > >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <[email protected]> >> >wrote: >> > >> >> We can keep the Main classes in the share lib, but have the action >> >> executors package the main classes for shipping it to >> >> Launcher (basically again having the method getLauncherClasses() in >> >>action >> >> executors). Also the oozie-sharelib.jar containing main classes >> >> should be part of web-app. >> >> This will improve the building/testing env as classes are in isolated >> >> sharelibs but will make the share lib optional as main classes are part >> >>of >> >> oozie-server. >> >> Can we do this change to trunk as this is blocking our QE's to pick >> >>builds >> >> due to share lib incompatibility? >> >> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with >> Hcat >> >> integration, >> >> there can be multiple overridable implementation of URIHandlers that >> can >> >> be shipped to launcher. >> >> We can do the same for other main classes if required. >> >> >> >> >> >> Thanks, >> >> Virag >> >> >> >> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <[email protected]> wrote: >> >> >> >> >That's true, its primarily a building/testing improvement. And from a >> >> >design perspective, it could be considered "cleaner" to not have the >> >> >launcher jar. >> >> > >> >> >Your idea sounds like a good compromise between backwards >> compatibility >> >> >and >> >> >improving the building/testing env. How would we do that? >> >> > >> >> >As a side note, there's also a new feature with the Main classes being >> >>in >> >> >the sharelib; users can now override the Main class that Oozie will >> use >> >> >for >> >> >the action because its no longer in the launcher jar. I think this >> >>could >> >> >still be a useful feature, but I'm guessing that if we do the >> >>compromise, >> >> >this would go away? In any case, OOZIE-1318 is to make the Main >> >> >overridable with a config, which is probably cleaner and more flexible >> >> >than >> >> >replacing a jar file. >> >> > >> >> >thanks >> >> >- Robert >> >> > >> >> > >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <[email protected]> >> >> >wrote: >> >> > >> >> >> That is a good suggestion for preserving compatibility, Robert. >> >> >> >> >> >> Another question is what is the main purpose of OOZIE-1311? From >> JIRA >> >> >> description, the primary >> >> >> advantage of moving Main classes from core to share lib is it can >> >>help >> >> >>in >> >> >> preventing dependencies conflicts >> >> >> (different versions of antlr's for pig, hive). However, that problem >> >> >>only >> >> >> exists while building/testing Oozie. >> >> >> If that is the case, main classes can be moved to share lib but they >> >>can >> >> >> be bundled in oozie-server itself. >> >> >> In that case, the deployment doesn't change and share lib can still >> >>be >> >> >> optional >> >> >> as oozie server would ship those classes (same as what was happening >> >> >> before). And, >> >> >> advantage of 1311 for building/testing will be retained >> >> >> >> >> >> The main concern is having share lib as a required installation will >> >> >> complicate deployment. >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't >> >>provide >> >> >> inbuilt support for that. >> >> >> Even though end-users wouldn't notice the change, it will be a big >> >> >>change >> >> >> for system admins installing oozie >> >> >> So even though share lib is nice feature to have, keeping it >> optional >> >> >> seems better to me as there are other ways >> >> >> Of including jars and which were introduced much before share lib. >> >> >> >> >> >> Thanks, >> >> >> Virag >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <[email protected]> wrote: >> >> >> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for >> >>existing >> >> >> >apps where the sharelib isn't used. Assuming the sharelib is >> >> >>installed, >> >> >> >if >> >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie >> >>would >> >> >> >only include the oozie-*.jar from the sharelib (the one with the >> >>Main) >> >> >>in >> >> >> >it; if its set to true, then Oozie would include all of the jars in >> >>the >> >> >> >sharelib. This would allow users to still use their own lib dir >> >>while >> >> >> >still getting the Main into the classpath. >> >> >> > >> >> >> >- Robert >> >> >> > >> >> >> > >> >> >> > >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur >> >> >> ><[email protected]>wrote: >> >> >> > >> >> >> >> virag, if your oozie servers dont have sharelibs currently >> >>installed >> >> >> >>then >> >> >> >> unstalling sharelibs as part of an update would not break ant >> >>running >> >> >> >>app. >> >> >> >> regarding the requirement of enabling the sharelibs to have the >> >> >>mains, >> >> >> >>you >> >> >> >> have a point. we need to figure out how to get those in the >> >> >>classpath if >> >> >> >> the user is not using the sharelibs. i'd suggest we leave this >> >> >> >> patch in trunk for now and we move it to a release branch once we >> >> >>sorted >> >> >> >> out a way of solving this incompatibility. >> >> >> >> >> >> >> >> thanks >> >> >> >> >> >> >> >> Alejandro >> >> >> >> (phone typing) >> >> >> >> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <[email protected]> >> >> >>wrote: >> >> >> >> >> >> >> >> > >> >> >> >> > Tucu, Thanks for the reply and explanation. >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible >> >>option >> >> >>as >> >> >> >>we >> >> >> >> > cannot expect all the running actions to complete before Oozie >> >> >> >>restart. >> >> >> >> > Hot upgrade is a good option, but we will need time to >> >>implement it >> >> >> >>as we >> >> >> >> > currently don't have sharelib as part of our deployment. >> >> >> >> > So can we deprecate the refactoring of launcher classes in this >> >> >> >>release? >> >> >> >> > Also, all the launcher classes are only added to distributed >> >>cache >> >> >>if >> >> >> >>the >> >> >> >> > workflow is configured to use system lib path >> >> >> >> > (oozie.use.system.libpath=true) >> >> >> >> > We shouldn't mandate the workflow to have this property as they >> >>are >> >> >> >> > oozie's launcher classes. >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Virag >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <[email protected]> >> >>wrote: >> >> >> >> > >> >> >> >> >> Virag, >> >> >> >> >> >> >> >> >> >> On sharelib being required, yes you are correct. For this we >> >> >>should: >> >> >> >> >> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the >> >> >>sharelib >> >> >> >>is >> >> >> >> >> REQUIRED. >> >> >> >> >> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir >> >>exists, >> >> >> >>else >> >> >> >> >> fail >> >> >> >> >> to start. >> >> >> >> >> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. >> >> >>Depending on >> >> >> >> the >> >> >> >> >> type of upgrade this may be an issue. >> >> >> >> >> >> >> >> >> >> * If you are upgrading an oozie server fix that does not >> change >> >> >>the >> >> >> >> >> sharelib files, this is not an issue and you can just shutdown >> >>the >> >> >> >>oozie >> >> >> >> >> server with in-flight jobs. >> >> >> >> >> >> >> >> >> >> * If you are upgrading an oozie server fix that involves >> >>sharelib >> >> >> >>files >> >> >> >> >> changes, then you have 2 options: >> >> >> >> >> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait >> >> >>till >> >> >> >>all >> >> >> >> >> running actions end, then shutdown oozie server, upgrade oozie >> >> >>server >> >> >> >> and >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In >> this >> >> >>case >> >> >> >>all >> >> >> >> >> new WF actions will use the new sharelib. >> >> >> >> >> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml >> >> >>sharelib >> >> >> >> >> location to point to a new directory. upgrade the oozie >> server. >> >> >> >>install >> >> >> >> >> the >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does >> not >> >> >> >>exist). >> >> >> >> >> start the oozie server. In this case all running WF actions >> >>will >> >> >> >> continue >> >> >> >> >> running with no issues as the JARs in the distributed cache >> >>have >> >> >>not >> >> >> >> been >> >> >> >> >> touch. All new WF actions will start using the new sharelib. >> >> >> >> >> >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced by >> >> >> >>requiring >> >> >> >> >> sharelib, it is required if you have applications that use >> >> >>sharelib >> >> >> >> today. >> >> >> >> >> >> >> >> >> >> Does this address your concerns? >> >> >> >> >> >> >> >> >> >> Thanks. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari >> >> >><[email protected]> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >>> Hi, >> >> >> >> >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to move all >> >>the >> >> >> >> >>> launcher >> >> >> >> >>> classes like PigMain, HiveMain etc. to their respective >> >> >>sharelibs. >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before >> >>shared >> >> >>lib >> >> >> >> was >> >> >> >> >>> optional as users could bundle jars with their workflow >> >> >>application. >> >> >> >> >>> So always requiring shared lib seems to introduce 2 problems: >> >> >> >> >>> >> >> >> >> >>> 1. The current deployments which don't use action shared >> lib >> >> >>will >> >> >> >> >>> fail. >> >> >> >> >>> So, probably we should deprecate the current behavior. >> >> >> >> >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if >> >>the >> >> >> >>files >> >> >> >> >>> in >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So, >> >>when >> >> >> >>Oozie >> >> >> >> >>> is >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of >> >> >> >> >>> deployment, hadoop will fail the existing jobs >> >>for >> >> >> >>which >> >> >> >> >>> the >> >> >> >> >>> timestamp of the file on hdfs doesn't match the timestamp of >> >>its >> >> >> >>copy >> >> >> >> >>> in >> >> >> >> >>> the job's DC. >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> Thanks, >> >> >> >> >>> Virag >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >
