[ https://issues.apache.org/jira/browse/HADOOP-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022336#comment-16022336 ]
Yongjun Zhang commented on HADOOP-14407: ---------------------------------------- Welcome [~omkarksa]. My bad, did not catch an issue in your branch-2 patch in time. {code} ------------------------------------------------------ Running org.apache.hadoop.tools.TestDistCpOptions Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec <<< FAILURE! - in org.apache.hadoop.tools.TestDistCpOptions testToString(org.apache.hadoop.tools.TestDistCpOptions) Time elapsed: 0.018 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<..., filtersFile='null'[]}> but was:<..., filtersFile='null'[, blocksPerChunk=0, copyBufferSize=8192]}> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.hadoop.tools.TestDistCpOptions.testToString(TestDistCpOptions.java:317) {code} TestDistCpOptions.java is somehow missing from the branch-2 patch, would you please create a new jira for branch-2 only and submit the patch asap? Thanks. > DistCp - Introduce a configurable copy buffer size > -------------------------------------------------- > > Key: HADOOP-14407 > URL: https://issues.apache.org/jira/browse/HADOOP-14407 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp > Affects Versions: 2.9.0 > Reporter: Omkar Aradhya K S > Assignee: Omkar Aradhya K S > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14407.001.patch, HADOOP-14407.002.patch, > HADOOP-14407.002.patch, HADOOP-14407.003.patch, > HADOOP-14407.004.branch2.patch, HADOOP-14407.004.patch, > HADOOP-14407.004.patch, TotalTime-vs-CopyBufferSize.jpg > > > Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just > 8KB. We have noticed in our performance tests that with bigger buffer sizes > we saw upto ~3x performance boost. Hence, making the copy buffer size a > configurable setting via the new parameter <copybuffersize>. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org