[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219646#comment-15219646 ] Lin Yiqun commented on HDFS-10234: -- Upload a simple patch from me. I added the log output info for copying and and creating directories. But it seems there was no place indicated that the file is deleted in class {{CopyMapper}}. Can this patch satisfied you. If I was missing something, you can tell me. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov > Attachments: HDFS-10234.001.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219696#comment-15219696 ] Hadoop QA commented on HDFS-10234: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 23s {color} | {color:green} hadoop-distcp in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 16s {color} | {color:green} hadoop-distcp in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 6s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:fbe3e86 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12796266/HDFS-10234.001.patch | | JIRA Issue | HDFS-10234 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 52293d3b477c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221240#comment-15221240 ] Konstantin Shaposhnikov commented on HDFS-10234: I guess my problem description was not very precise. The log file that I am interested in is created when `-log logDir` option is specified. This file is written using {{context.write}} (e.g. {{context.write(null, new Text("SKIP: " + sourceCurrStatus.getPath()));}}). Currently this file contains only SKIP entries making it almost useless. There is no need to make logging on INFO level more verbose as it is already too verbose. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Lin Yiqun > Attachments: HDFS-10234.001.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221242#comment-15221242 ] Konstantin Shaposhnikov commented on HDFS-10234: Another way to make the job output more useful is to have separate counters for copied files and directories. Currently even if a directory exists on target COPY counter is incremented. So if I run distcp with -update option it still reports large number in COPY counter even though the number of copied files is 0. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Lin Yiqun > Attachments: HDFS-10234.001.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225784#comment-15225784 ] Lin Yiqun commented on HDFS-10234: -- Thanks [~k.shaposhni...@gmail.com] for concrete descriptions. Update the latest patch for addressing your comments. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Lin Yiqun > Attachments: HDFS-10234.001.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225831#comment-15225831 ] Hadoop QA commented on HDFS-10234: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 43s {color} | {color:red} hadoop-distcp in the patch failed with JDK v1.8.0_77. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s {color} | {color:green} hadoop-extras in the patch passed with JDK v1.8.0_77. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 37s {color} | {color:red} hadoop-distcp in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s {color} | {color:green} hadoop-extras in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black}
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139334#comment-16139334 ] Xiaoyu Yao commented on HDFS-10234: --- [~linyiqun], thanks for the patch. I take a look at the v2 patch here are my comments: 1. We need replace the COPY count update in createTargetDirsWithRetry with a DIR_COPY count update. The change to update DIR_COPY counter in map() can be removed after that. Without that, [~k.shaposhni...@gmail.com]'s earlier comment are not fully addressed. {code} @@ -260,7 +268,7 @@ private void createTargetDirsWithRetry(String description, } catch (Exception e) { throw new IOException("mkdir failed for " + target, e); } -incrementCounter(context, Counter.COPY, 1); +incrementCounter(context, Counter.DIR_COPY, 1); } {code} 2. Can we include both the source (path, size) and destination (path, size) in the SKIP/COPY log inside map()? The information is available from sourceCurrStatus and targetStatus there. This way, many applications can just parse the distcp log offline to get information without adding extra load on namenode. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139336#comment-16139336 ] Hadoop QA commented on HDFS-10234: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color} | {color:red} HDFS-10234 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-10234 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12796999/HDFS-10234.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20831/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139841#comment-16139841 ] Yiqun Lin commented on HDFS-10234: -- Thanks for the review, Xiaoyu! Seems this JIRA hasn't been updated more than one year, I will spend some time to look into this again recent days, :). Thanks. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143341#comment-16143341 ] Hadoop QA commented on HDFS-10234: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated 1 new + 139 unchanged - 0 fixed = 140 total (was 139) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 54s{color} | {color:green} hadoop-distcp in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 32m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-10234 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883989/HDFS-10234.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ae06e9330b77 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ae7abad | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20885/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20885/testReport/ | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20885/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee:
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145775#comment-16145775 ] Xiaoyu Yao commented on HDFS-10234: --- Thanks [~linyiqun] for the update. The patch v3 looks pretty good to me. I just have few comments below: DistCpOptionSwitch.java Line 125: NIT: "Log additional info (path, size) in the SKIP/COPY log." CopyMapper.java Line 86: NIT: verboseWrite -> verboseLog Line 110: Can we check and ensure verboseWrite can only be used with -log option? DistCp.md.vm Line 223: `-v` | Log additional info (path, size) in the SKIP/COPY log | This option can only be used with -log option. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch, > HDFS-10234.003.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147009#comment-16147009 ] Yiqun Lin commented on HDFS-10234: -- Thanks for the review, [~xyao]! Attach the updated patch to address your comments. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch, > HDFS-10234.003.patch, HDFS-10234.004.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147051#comment-16147051 ] Hadoop QA commented on HDFS-10234: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 15s{color} | {color:red} hadoop-distcp in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 35m 7s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.tools.TestDistCpOptions | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-10234 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884452/HDFS-10234.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 4bfdb7a09512 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 200b113 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20923/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20923/testReport/ | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20923/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attac
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147119#comment-16147119 ] Hadoop QA commented on HDFS-10234: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 9s{color} | {color:green} hadoop-distcp in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 36m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-10234 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884459/HDFS-10234.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux b36ed5c9b44c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 200b113 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20925/testReport/ | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20925/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch, > HDFS-10234.003.patch, HDFS-10234.004.patch, HDFS-10234.005.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contai
[jira] [Commented] (HDFS-10234) DistCp log output should contain copied and deleted files and directories
[ https://issues.apache.org/jira/browse/HDFS-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147553#comment-16147553 ] Xiaoyu Yao commented on HDFS-10234: --- Thanks [~linyiqun] for the update. Patch v5 looks good to me. +1. I will hold off the commit until Friday (9/1) in case [~k.shaposhni...@gmail.com] and/or other folks may have additional comments. > DistCp log output should contain copied and deleted files and directories > - > > Key: HDFS-10234 > URL: https://issues.apache.org/jira/browse/HDFS-10234 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 2.7.1 >Reporter: Konstantin Shaposhnikov >Assignee: Yiqun Lin > Attachments: HDFS-10234.001.patch, HDFS-10234.002.patch, > HDFS-10234.003.patch, HDFS-10234.004.patch, HDFS-10234.005.patch > > > DistCp log output (specified via {{-log}} command line option) currently > contains only skipped and failed (when failures are ignored via {{-i}}) files. > It will be more useful if it also contains copied and deleted files and > created directories. > This should be fixed in > https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org