> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many
> > people launch hbase jobs from java action and instead of each user adding
> > killChildJobs doing it in JavaActionExecutor should be good. Currently the
> > configuration is not passed properly to distcp action. For killing child
> > jobs in distcp to work, we need to fix that or pass the configuration as
> > arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with
> > that tag. I think we can pass it as arg for now.
>
> Robert Kanter wrote:
> The reason I didn't add the killChildJobs to the Distcp action is that we
> don't have a "DistcpMain" to put it in. I didn't want to put the
> killChildJobs in the calling class LauncherMapper because (a) we handle this
> differently for the MR action and (b) someone writing a custom action (or us
> in a future action) may also want to handle this differently. I guess I
> could write a wrapper "DistcpMain" to call killChildJobs...
>
> For the Java and Shell actions, they don't necessarily launch any child
> jobs. And in general, we don't implement some of the "Oozie magic" stuff for
> the Java and Shell actions, leaving it up to the user instead (for example,
> propagating the action's <configuration> or the delegation token). The Java
> action also has the same problem as the Distcp action, in that there's no
> "JavaMain". What if I added documentation on how to call killChildJobs so
> the user can do this in their Main?
>
> Rohini Palaniswamy wrote:
> Many users launch child jobs in their java actions (especially with
> hbase). Would prefer as much less work for user as possible. If you are
> writing a DistcpMain, you can just write a JavaMain that calls killChildJobs
> and make DistcpMain just extend that. That way it will just work without
> users having to bother about it and writing different code for different
> versions of hadoop.
>
> Robert Kanter wrote:
> I can add a wrapper for Java and Distcp that does the killing. Though
> for the Java action, it will still be up to the user to propagate the
> actionConf prepared by Oozie, or at least just set the tag property.
>
> Rohini Palaniswamy wrote:
> Sounds good and thanks. Most of them usually propagate the actionConf and
> would benefit from this.
>
> Robert Kanter wrote:
> Getting this to work with DistCp is trickier than I thought. I tried
> passing the tags property as a -D argument, but that didn't work (in fact, it
> caused an error). Looking at DistCp's code, we could try to invoke it
> programmatically from a "DistCpMain" instead of JavaMain; this would allow us
> to pass it a Configuration. However, DistCp v1 (Hadoop 1) and v2 (Hadoop
> 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an
> additional argument), so the "DistCpMain" would have to use Reflection and
> check which constructor is available. MAPREDUCE-5081 did backport v2 to
> Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>
> So we have a few options:
> (1) Don't support the tags with DistCp
> (2) Use a "DistCpMain"
> (2a) Use reflection to call the appropriate constructor and run it
> programmatically
> (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of
> Hadoop
>
> I think we should do (1) for now so we can get the tags stuff in and
> create a follow-up JIRA to do (2b), which will require a lot of additional
> testing.
>
> Thoughts?
>
> Rohini Palaniswamy wrote:
> Option 3:
> <arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information
> is currently passed to Distcp from the distcp action.
> LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments
> handle the main method arguments. We need to introduce it somewhere there for
> Distcp. But it is hacky and don't like it much. Just pointing out as an
> option.
>
> Your option 2 is cleaner and like it better. We definitely need to fix
> DistcpMain to call the constructor instead of the main() method to pass the
> JobConf. Users have been passing even the queue information as above because
> the distcp job does not get the action configuration. We had this reported as
> an issue, but have always gone with the <arg> workaround. So option 2 will
> fix more issues. We can definitely do (2a). (2b) of course can be a different
> jira.
>
> Rohini Palaniswamy wrote:
> Actually with your current patch you can easily do Option 3 (passing
> -Dmapreduce.job.tags=xxx as main args) with DistcpMain as you would be
> invoking the Distcp main class from there. Was that what you tried and did
> not work?
>
> Robert Kanter wrote:
> I did try option 3, using LauncherMapperHelper.setupMainArguments. It
> gave an error about not understanding the -D argument. I think that only
> works from the hadoop CLI, which parses out the -D args.
>
> I can try 2a, but I think this will be very brittle. The main() in each
> DistCp is different (e.g. v2 has a shutdown hook it configures). So while I
> can try to duplicate each main() in the reflection, it can easily break if
> DistCp changes or if someone is using a different version of DistCp. I think
> we should wait and do 1 and do 2b later.
How does <arg>-Dmapred.job.queue.name=${queueName}</arg> get passed to distcp
then?
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------
On March 4, 2014, 9:48 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
>
> (Updated March 4, 2014, 9:48 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1722
> https://issues.apache.org/jira/browse/OOZIE-1722
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1722
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java
> fcf69e7
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> 68d77a8
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
> 07de214
>
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> 7841076
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7
> core/src/test/java/org/apache/oozie/service/TestShareLibService.java
> e7d7a26
> hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/pom.xml 35875c8
> pom.xml 1223ec2
>
> sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java
> 25b1a23
> sharelib/hive/pom.xml e355d97
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java
> 8fa5ec8
> sharelib/oozie/pom.xml c70582d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java
> PRE-CREATION
>
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
> b5fa4a1
>
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java
> 6230e64
> sharelib/pig/pom.xml a2bbeaf
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java
> 9c1e659
> sharelib/sqoop/pom.xml a962565
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java
> e45c8b2
>
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/18653/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
