[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2015-11-22 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--
Status: Open  (was: Patch Available)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2015-05-05 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated MAPREDUCE-5549:

Labels: BB2015-05-TBR  (was: )

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2015-03-11 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated MAPREDUCE-5549:

Target Version/s: 2.6.0, 3.0.0  (was: 3.0.0, 2.6.0)
  Status: Open  (was: Patch Available)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2015-03-11 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated MAPREDUCE-5549:

Target Version/s: 2.6.0, 3.0.0  (was: 3.0.0, 2.6.0)
  Status: Patch Available  (was: Open)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2014-07-06 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-5549:


Target Version/s: 3.0.0, 2.6.0  (was: 3.0.0)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2014-07-03 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Status: Patch Available  (was: Open)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2014-07-03 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Status: Open  (was: Patch Available)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2014-07-03 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Attachment: MAPREDUCE-5549-002.patch

patch rebased against trunk; no other changes

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2013-10-02 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Target Version/s: 3.0.0, 2.3.0
  Status: Patch Available  (was: Open)

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2013-10-02 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Attachment: MAPREDUCE-5549-001.patch

Patch with some changes from the original proposal

# check for {{job.isSuccess()}} performed in try/catch block, so that IOEs get 
caught.
# added new {{JOB_FAILED}} value; lets scripts & people interpret failure 
better 
I also took the opportunity to review the run method and patched the catch of 
Throwable in the parse phase to print Exception.toString, not 
Exception.getMessage, for the usual reasons.

Tests: no, none

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
> Attachments: MAPREDUCE-5549-001.patch
>
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (MAPREDUCE-5549) distcp app should fail if m/r job fails

2013-10-02 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-5549:
--

Affects Version/s: 3.0.0

> distcp app should fail if m/r job fails
> ---
>
> Key: MAPREDUCE-5549
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp, mrv2
>Affects Versions: 3.0.0
>Reporter: David Rosenstrauch
>
> I run distcpv2 in a scripted manner.  The script checks if the distcp step 
> fails and, if so, aborts the rest of the script.  However, I ran into an 
> issue today where the distcp job failed, but my calling script went on its 
> merry way.
> Digging into the code a bit more (at 
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
>  I think I see the issue:  the distcp app is not returning an error exit code 
> to the shell when the distcp job fails.  This is a big problem, IMO, as it 
> prevents distcp from being successfully used in a scripted environment.  IMO, 
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> try {
>   execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
>   }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
>   public int run(String[] argv) {
> //...
> Job job = null;
> try {
>   job = execute();
> } catch (InvalidInputException e) {
>   LOG.error("Invalid input: ", e);
>   return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
>   LOG.error("Duplicate files in input path: ", e);
>   return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
>   LOG.error("Exception encountered ", e);
>   return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
>   return DistCpConstants.SUCCESS;
> }
> else {
>   return DistCpConstants.UNKNOWN_ERROR;
> }
>   }
> //...
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)