[jira] [Comment Edited] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403297#comment-15403297 ] Frederick Tucker edited comment on MAPREDUCE-6734 at 8/30/16 11:11 PM: --- I'd like to kindly ask for a code review to this patch was (Author: fctucker): I'd like to kindly ask again for a code review to this patch > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Critical > Labels: distcp, newbie, patch > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Tucker updated MAPREDUCE-6734: Comment: was deleted (was: I am requesting a code review. "Please add a comment that asks for a code review following our code review checklist.") > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Critical > Labels: distcp, newbie, patch > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6454) MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449169#comment-15449169 ] Allen Wittenauer commented on MAPREDUCE-6454: - This patch makes a huge assumption about the shell scripts that is no longer true in trunk (and my attempts to teach bash in this jira failed). In the end, users who use hadoop, hdfs, yarn, and mapred commands in their code will discover they don't have distributed cache jars in their classpath. > MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache. > > > Key: MAPREDUCE-6454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6454 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Fix For: 2.7.2, 2.6.2 > > Attachments: MAPREDUCE-6454-v2.1.patch, MAPREDUCE-6454-v2.patch, > MAPREDUCE-6454-v3.1.patch, MAPREDUCE-6454-v3.patch, MAPREDUCE-6454.patch > > > We already set lib jars on distributed-cache to CLASSPATH. However, in some > corner cases (like: MR local mode, Hive Map side local join, etc.), we need > these jars on HADOOP_CLASSPATH so hadoop scripts can take it in launching > runjar process. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4784) TestRecovery occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449145#comment-15449145
]
Hudson commented on MAPREDUCE-4784:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10374 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10374/])
MAPREDUCE-4784. TestRecovery occasionally fails. Contributed by Haibo (jlowe:
rev af508605a9edc126c170160291dbc2fe58b66dea)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
> TestRecovery occasionally fails
> ---
>
> Key: MAPREDUCE-4784
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4784
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: mapreduce4784.001.patch
>
>
> TestRecovery is occasionally failing with this error:
> {noformat}
> testCrashed(org.apache.hadoop.mapreduce.v2.app.TestRecovery): TaskAttempt
> state is not correct (timedout) expected: but was:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4784) TestRecovery occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4784:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.7.4
2.8.0
Status: Resolved (was: Patch Available)
Thanks, [~haibochen]! I committed this to trunk, branch-2, branch-2.8, and
branch-2.7.
> TestRecovery occasionally fails
> ---
>
> Key: MAPREDUCE-4784
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4784
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: mapreduce4784.001.patch
>
>
> TestRecovery is occasionally failing with this error:
> {noformat}
> testCrashed(org.apache.hadoop.mapreduce.v2.app.TestRecovery): TaskAttempt
> state is not correct (timedout) expected: but was:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4784) TestRecovery occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449085#comment-15449085
]
Jason Lowe commented on MAPREDUCE-4784:
---
+1 lgtm. Committing this.
> TestRecovery occasionally fails
> ---
>
> Key: MAPREDUCE-4784
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4784
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Attachments: mapreduce4784.001.patch
>
>
> TestRecovery is occasionally failing with this error:
> {noformat}
> testCrashed(org.apache.hadoop.mapreduce.v2.app.TestRecovery): TaskAttempt
> state is not correct (timedout) expected: but was:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6454) MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448842#comment-15448842 ] Junping Du commented on MAPREDUCE-6454: --- [~andrew.wang], Yes. this and MAPREDUCE-6619 only for branch-2 now. For trunk, Allen want to do something different but I am not sure the progress there. May be we should consider to port this and MAPREDUCE-6619 to trunk as well? > MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache. > > > Key: MAPREDUCE-6454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6454 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Fix For: 2.7.2, 2.6.2 > > Attachments: MAPREDUCE-6454-v2.1.patch, MAPREDUCE-6454-v2.patch, > MAPREDUCE-6454-v3.1.patch, MAPREDUCE-6454-v3.patch, MAPREDUCE-6454.patch > > > We already set lib jars on distributed-cache to CLASSPATH. However, in some > corner cases (like: MR local mode, Hive Map side local join, etc.), we need > these jars on HADOOP_CLASSPATH so hadoop scripts can take it in launching > runjar process. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
