[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607531#comment-14607531 ] Hadoop QA commented on MAPREDUCE-5549: -- \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / d3797f9 | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5862/console | This message was automatically generated. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357822#comment-14357822 ] Hadoop QA commented on MAPREDUCE-5549: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch against trunk revision 7a346bc. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5282//console This message is automatically generated. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054446#comment-14054446 ] Gera Shegalov commented on MAPREDUCE-5549: -- Ravi the base class is {{Throwable}}, therefore is not sufficient to catch an exception. I find the way it's handled via {{setDefaultUncaughtExceptionHandler}} more elegant. There are a bunch of examples in Hadoop including MapReduce. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053377#comment-14053377 ] Ravi Prakash commented on MAPREDUCE-5549: - Just saw your comment Gera. Do you have a suggestion for how an exception in the main thread should be handled differently than it is now? From what I can see, Exception is being caught, LOG.error("Couldn't complete DistCp operation: ", e); and exitCode is being returned. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053375#comment-14053375 ] Ravi Prakash commented on MAPREDUCE-5549: - Thanks a lot for the patch David and Steve. The return code should be -5 (in DistCpConstants.java) because -3 is already being used by ACLS_NOT_SUPPORTED. Other than that, the patch looks good and I'll be happy to commit it once that change is made. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053244#comment-14053244 ] Gera Shegalov commented on MAPREDUCE-5549: -- We should also consider using setDefaultUncaughtExceptionHandler at least for the main thread to exit with another error code. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051373#comment-14051373 ] Hadoop QA commented on MAPREDUCE-5549: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-distcp. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4709//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4709//console This message is automatically generated. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784312#comment-13784312 ] Ravi Prakash commented on MAPREDUCE-5549: - +1. Looks good to me. Thanks Steve! > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784010#comment-13784010 ] Hadoop QA commented on MAPREDUCE-5549: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606282/MAPREDUCE-5549-001.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-distcp. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4079//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4079//console This message is automatically generated. > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Affects Versions: 3.0.0 >Reporter: David Rosenstrauch > Attachments: MAPREDUCE-5549-001.patch > > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783396#comment-13783396 ] Ravi Prakash commented on MAPREDUCE-5549: - This seems like trunk and yes, I agree that if the job fails, then the job should return a proper error code > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Reporter: David Rosenstrauch > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782800#comment-13782800 ] Steve Loughran commented on MAPREDUCE-5549: --- usual questions: which versions of hadoop, and can you supply the change as a .patch file, either from diff or git diff. thanks > distcp app should fail if m/r job fails > --- > > Key: MAPREDUCE-5549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, mrv2 >Reporter: David Rosenstrauch > > I run distcpv2 in a scripted manner. The script checks if the distcp step > fails and, if so, aborts the rest of the script. However, I ran into an > issue today where the distcp job failed, but my calling script went on its > merry way. > Digging into the code a bit more (at > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), > I think I see the issue: the distcp app is not returning an error exit code > to the shell when the distcp job fails. This is a big problem, IMO, as it > prevents distcp from being successfully used in a scripted environment. IMO, > the code should change like so: > Before: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > try { > execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > return DistCpConstants.SUCCESS; > } > //... > {code} > After: > {code:title=org.apache.hadoop.tools.DistCp.java} > //... > public int run(String[] argv) { > //... > Job job = null; > try { > job = execute(); > } catch (InvalidInputException e) { > LOG.error("Invalid input: ", e); > return DistCpConstants.INVALID_ARGUMENT; > } catch (DuplicateFileException e) { > LOG.error("Duplicate files in input path: ", e); > return DistCpConstants.DUPLICATE_INPUT; > } catch (Exception e) { > LOG.error("Exception encountered ", e); > return DistCpConstants.UNKNOWN_ERROR; > } > if (job.isSuccessful()) { > return DistCpConstants.SUCCESS; > } > else { > return DistCpConstants.UNKNOWN_ERROR; > } > } > //... > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)