> On Nov. 12, 2013, 8:19 p.m., Robert Kanter wrote: > > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, > > line 289 > > <https://reviews.apache.org/r/14741/diff/9/?file=381242#file381242line289> > > > > Instead of using a synchronized block here, can you use the Locks > > service? This way, when Oozie HA is enabled, it will automatically use the > > distributed locks so multiple Oozie servers won't try to update the > > sharelib/launcherlib at the same time. You can use a lock named > > "org.apache.oozie.service.ShareLibService". > > > > We should also put this below where the other synchronized block is. > > > > There are probably other places where this could be a problem too, like > > purging the sharelib. I think that simply surrounding it with a lock from > > the Locks service should take care of these cases. > > > > Also, I assume we only want one server to deal with purging the > > sharelib, right? If so, you can use the #isFirstServer() method from the > > JobsConcurrencyService to check if the server should be doing the purge. > > It always returns true for non-HA and only returns true for one server with > > HA.
synchronized is used for caching file from DFS. There is no update here. It used in scenario where caching of sharelib at startup failed. At the first job submission we try to cache sharelib. We synchronized the block with double check to take care of multiple job submission at same time. We might have to think for system launcher jar. When SHIP_LAUNCHER_JAR=true server tries to copy jars from local to DFS. I believe copy of system launcher jar should be ok. With current implementation system launcher jar are always copied to DFS with new timestamp directory. Each server will be using his own version of system share lib. We might have to think of purging, currently server delete anything older than 7 days.In case of HA, restart of 1 server might delete system share lib of long running other Oozie server. Yup, only one server should purge the sharelib, we will use isFirstServer() for that. - Purshotam ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14741/#review28752 ----------------------------------------------------------- On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14741/ > ----------------------------------------------------------- > > (Updated Nov. 12, 2013, 7:56 p.m.) > > > Review request for oozie. > > > Bugs: OOZIE-1584 > https://issues.apache.org/jira/browse/OOZIE-1584 > > > Repository: oozie > > > Description > ------- > > Supported features. > ----------------- > 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at > server start. > > 2. Admin copies sharelib to new timestamp directory and issues invalidate > command to Oozie server. Oozie picks latest shared lib. > > 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest > sharelib jar pushed by admin(based on timestamp). > > 4. Sharelib mapping files. > Sharelib mapping file can be also configured.Configured file is a key value > mapping, where key will be the action and value is DFS directory of sharelib > files for action. > This can be configured in oozie-site.xml as : > <property> > <name>oozie.service.ShareLibService.mapping.file</name> > <value></value> > <description> > Sharelib mapping files contains list of key=value, > where key is the action key and value is the DFS location of > sharelib files. > </description> > </property> > > -------------------- > DFS after generating launcher jar by ooozie server and sharelib by cli > > makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls > /user/purushah/share/lib/ > /user/purushah/share/lib/launcher_20131017092254 > /user/purushah/share/lib/launcher_20131017093814 > /user/purushah/share/lib/launcher_20131017094652 > /user/purushah/share/lib/launcher_20131017094836 > /user/purushah/share/lib/launcher_20131017095549 > /user/purushah/share/lib/lib_20131017092806 > makebag-lm:example purushah$ > > > Purging. > ------------- > There are two set( launcher_ for launcher jars and Lib_ for shared lib) of > directory and purging happens for both at startup. > Anything older than ( defined in property) is purged. > We always keep 2 set of directory, purging happens if number of sharelib > directory > 2. > > > Multiple version. > ------------------- > To support multiple version, sharelib will be in below format. > > Sharelib will cache all versions and return list of dfs files for a particular > version. > > makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls > /user/purushah/share/lib/ > Found 15 items > /user/purushah/share/lib/launcher_20131017092254 > /user/purushah/share/lib/launcher_20131017093814 > /user/purushah/share/lib/launcher_20131017094652 > /user/purushah/share/lib/launcher_20131017094836 > /user/purushah/share/lib/launcher_20131017095549 > /user/purushah/share/lib/lib_20131017092806/pig/... > /user/purushah/share/lib/lib_20131017092806/pig_9/... > /user/purushah/share/lib/lib_20131017092806/pig_10/... > makebag-lm:example purushah$ > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki > 1532127 > > http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java > 1532127 > > Diff: https://reviews.apache.org/r/14741/diff/ > > > Testing > ------- > > > Thanks, > > Purshotam Shah > >
