I think I follow what you're saying; assuming everything works correctly, I think that sounds good. Can you make a JIRA with that and put up a patch; that might be easier to see how this would work.
thanks - Robert On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <[email protected]> wrote: > We can keep the Main classes in the share lib, but have the action > executors package the main classes for shipping it to > Launcher (basically again having the method getLauncherClasses() in action > executors). Also the oozie-sharelib.jar containing main classes > should be part of web-app. > This will improve the building/testing env as classes are in isolated > sharelibs but will make the share lib optional as main classes are part of > oozie-server. > Can we do this change to trunk as this is blocking our QE's to pick builds > due to share lib incompatibility? > > We can still do what oozie-1318 is trying to achieve. For, e.g with Hcat > integration, > there can be multiple overridable implementation of URIHandlers that can > be shipped to launcher. > We can do the same for other main classes if required. > > > Thanks, > Virag > > > On 4/24/13 10:18 AM, "Robert Kanter" <[email protected]> wrote: > > >That's true, its primarily a building/testing improvement. And from a > >design perspective, it could be considered "cleaner" to not have the > >launcher jar. > > > >Your idea sounds like a good compromise between backwards compatibility > >and > >improving the building/testing env. How would we do that? > > > >As a side note, there's also a new feature with the Main classes being in > >the sharelib; users can now override the Main class that Oozie will use > >for > >the action because its no longer in the launcher jar. I think this could > >still be a useful feature, but I'm guessing that if we do the compromise, > >this would go away? In any case, OOZIE-1318 is to make the Main > >overridable with a config, which is probably cleaner and more flexible > >than > >replacing a jar file. > > > >thanks > >- Robert > > > > > >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <[email protected]> > >wrote: > > > >> That is a good suggestion for preserving compatibility, Robert. > >> > >> Another question is what is the main purpose of OOZIE-1311? From JIRA > >> description, the primary > >> advantage of moving Main classes from core to share lib is it can help > >>in > >> preventing dependencies conflicts > >> (different versions of antlr's for pig, hive). However, that problem > >>only > >> exists while building/testing Oozie. > >> If that is the case, main classes can be moved to share lib but they can > >> be bundled in oozie-server itself. > >> In that case, the deployment doesn't change and share lib can still be > >> optional > >> as oozie server would ship those classes (same as what was happening > >> before). And, > >> advantage of 1311 for building/testing will be retained > >> > >> The main concern is having share lib as a required installation will > >> complicate deployment. > >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't provide > >> inbuilt support for that. > >> Even though end-users wouldn't notice the change, it will be a big > >>change > >> for system admins installing oozie > >> So even though share lib is nice feature to have, keeping it optional > >> seems better to me as there are other ways > >> Of including jars and which were introduced much before share lib. > >> > >> Thanks, > >> Virag > >> > >> > >> > >> > >> On 4/24/13 9:09 AM, "Robert Kanter" <[email protected]> wrote: > >> > >> >Alejandro had another suggestion to prevent breaking stuff for existing > >> >apps where the sharelib isn't used. Assuming the sharelib is > >>installed, > >> >if > >> >the user doesn't set oozie.use.system.libpath to true, then Oozie would > >> >only include the oozie-*.jar from the sharelib (the one with the Main) > >>in > >> >it; if its set to true, then Oozie would include all of the jars in the > >> >sharelib. This would allow users to still use their own lib dir while > >> >still getting the Main into the classpath. > >> > > >> >- Robert > >> > > >> > > >> > > >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > >> ><[email protected]>wrote: > >> > > >> >> virag, if your oozie servers dont have sharelibs currently installed > >> >>then > >> >> unstalling sharelibs as part of an update would not break ant running > >> >>app. > >> >> regarding the requirement of enabling the sharelibs to have the > >>mains, > >> >>you > >> >> have a point. we need to figure out how to get those in the > >>classpath if > >> >> the user is not using the sharelibs. i'd suggest we leave this > >> >> patch in trunk for now and we move it to a release branch once we > >>sorted > >> >> out a way of solving this incompatibility. > >> >> > >> >> thanks > >> >> > >> >> Alejandro > >> >> (phone typing) > >> >> > >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <[email protected]> > >>wrote: > >> >> > >> >> > > >> >> > Tucu, Thanks for the reply and explanation. > >> >> > For workflows running in Y!, cold upgrade is not a feasible option > >>as > >> >>we > >> >> > cannot expect all the running actions to complete before Oozie > >> >>restart. > >> >> > Hot upgrade is a good option, but we will need time to implement it > >> >>as we > >> >> > currently don't have sharelib as part of our deployment. > >> >> > So can we deprecate the refactoring of launcher classes in this > >> >>release? > >> >> > Also, all the launcher classes are only added to distributed cache > >>if > >> >>the > >> >> > workflow is configured to use system lib path > >> >> > (oozie.use.system.libpath=true) > >> >> > We shouldn't mandate the workflow to have this property as they are > >> >> > oozie's launcher classes. > >> >> > > >> >> > Thanks, > >> >> > Virag > >> >> > > >> >> > > >> >> > > >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <[email protected]> wrote: > >> >> > > >> >> >> Virag, > >> >> >> > >> >> >> On sharelib being required, yes you are correct. For this we > >>should: > >> >> >> > >> >> >> * Make 100% clear in the quick-start/install docs that the > >>sharelib > >> >>is > >> >> >> REQUIRED. > >> >> >> > >> >> >> * Add a check at Oozie startup to verify the sharelib dir exists, > >> >>else > >> >> >> fail > >> >> >> to start. > >> >> >> > >> >> >> On sharelib lib issue during upgrade for in-flight jobs. > >>Depending on > >> >> the > >> >> >> type of upgrade this may be an issue. > >> >> >> > >> >> >> * If you are upgrading an oozie server fix that does not change > >>the > >> >> >> sharelib files, this is not an issue and you can just shutdown the > >> >>oozie > >> >> >> server with in-flight jobs. > >> >> >> > >> >> >> * If you are upgrading an oozie server fix that involves sharelib > >> >>files > >> >> >> changes, then you have 2 options: > >> >> >> > >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait > >>till > >> >>all > >> >> >> running actions end, then shutdown oozie server, upgrade oozie > >>server > >> >> and > >> >> >> sharelib. Then restart oozie server and resume WF jobs. In this > >>case > >> >>all > >> >> >> new WF actions will use the new sharelib. > >> >> >> > >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml > >>sharelib > >> >> >> location to point to a new directory. upgrade the oozie server. > >> >>install > >> >> >> the > >> >> >> sharelib (will be a create as the sharelib dir in HDFS does not > >> >>exist). > >> >> >> start the oozie server. In this case all running WF actions will > >> >> continue > >> >> >> running with no issues as the JARs in the distributed cache have > >>not > >> >> been > >> >> >> touch. All new WF actions will start using the new sharelib. > >> >> >> > >> >> >> Note that this sharelib upgrade protocol is not introduced by > >> >>requiring > >> >> >> sharelib, it is required if you have applications that use > >>sharelib > >> >> today. > >> >> >> > >> >> >> Does this address your concerns? > >> >> >> > >> >> >> Thanks. > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari > >><[email protected]> > >> >> >> wrote: > >> >> >> > >> >> >>> Hi, > >> >> >>> > >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to move all the > >> >> >>> launcher > >> >> >>> classes like PigMain, HiveMain etc. to their respective > >>sharelibs. > >> >> >>> So, now shared lib is a mandatory deployment step. Before shared > >>lib > >> >> was > >> >> >>> optional as users could bundle jars with their workflow > >>application. > >> >> >>> So always requiring shared lib seems to introduce 2 problems: > >> >> >>> > >> >> >>> 1. The current deployments which don't use action shared lib > >>will > >> >> >>> fail. > >> >> >>> So, probably we should deprecate the current behavior. > >> >> >>> > >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if the > >> >>files > >> >> >>> in > >> >> >>> DC are updated on hdfs while the hadoop job is running. So, when > >> >>Oozie > >> >> >>> is > >> >> >>> restarted and shared lib is uploaded to hdfs as part of > >> >> >>> deployment, hadoop will fail the existing jobs for > >> >>which > >> >> >>> the > >> >> >>> timestamp of the file on hdfs doesn't match the timestamp of its > >> >>copy > >> >> >>> in > >> >> >>> the job's DC. > >> >> >>> > >> >> >>> > >> >> >>> Thanks, > >> >> >>> Virag > >> >> > > >> >> > >> > >> > >
