[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607531#comment-14607531
]
Hadoop QA commented on MAPREDUCE-5549:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d3797f9 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5862/console |
This message was automatically generated.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357822#comment-14357822
]
Hadoop QA commented on MAPREDUCE-5549:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch
against trunk revision 7a346bc.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5282//console
This message is automatically generated.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053375#comment-14053375
]
Ravi Prakash commented on MAPREDUCE-5549:
-
Thanks a lot for the patch David and Steve. The return code should be -5 (in
DistCpConstants.java) because -3 is already being used by ACLS_NOT_SUPPORTED.
Other than that, the patch looks good and I'll be happy to commit it once that
change is made.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053377#comment-14053377
]
Ravi Prakash commented on MAPREDUCE-5549:
-
Just saw your comment Gera. Do you have a suggestion for how an exception in
the main thread should be handled differently than it is now? From what I can
see, Exception is being caught, LOG.error(Couldn't complete DistCp operation:
, e); and exitCode is being returned.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054446#comment-14054446
]
Gera Shegalov commented on MAPREDUCE-5549:
--
Ravi the base class is {{Throwable}}, therefore is not sufficient to catch an
exception. I find the way it's handled via
{{setDefaultUncaughtExceptionHandler}} more elegant. There are a bunch of
examples in Hadoop including MapReduce.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053244#comment-14053244
]
Gera Shegalov commented on MAPREDUCE-5549:
--
We should also consider using setDefaultUncaughtExceptionHandler at least for
the main thread to exit with another error code.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051373#comment-14051373
]
Hadoop QA commented on MAPREDUCE-5549:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653802/MAPREDUCE-5549-002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4709//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4709//console
This message is automatically generated.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784010#comment-13784010
]
Hadoop QA commented on MAPREDUCE-5549:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606282/MAPREDUCE-5549-001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4079//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4079//console
This message is automatically generated.
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784312#comment-13784312
]
Ravi Prakash commented on MAPREDUCE-5549:
-
+1. Looks good to me. Thanks Steve!
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Affects Versions: 3.0.0
Reporter: David Rosenstrauch
Attachments: MAPREDUCE-5549-001.patch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782800#comment-13782800
]
Steve Loughran commented on MAPREDUCE-5549:
---
usual questions: which versions of hadoop, and can you supply the change as a
.patch file, either from diff or git diff.
thanks
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Reporter: David Rosenstrauch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5549) distcp app should fail if m/r job fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783396#comment-13783396
]
Ravi Prakash commented on MAPREDUCE-5549:
-
This seems like trunk and yes, I agree that if the job fails, then the job
should return a proper error code
distcp app should fail if m/r job fails
---
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Reporter: David Rosenstrauch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an
issue today where the distcp job failed, but my calling script went on its
merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
