[
https://issues.apache.org/jira/browse/HADOOP-17620?focusedWorklogId=581098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-581098
]
ASF GitHub Bot logged work on HADOOP-17620:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Apr/21 15:48
Start Date: 12/Apr/21 15:48
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2861:
URL: https://github.com/apache/hadoop/pull/2861#discussion_r611748559
##########
File path:
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -628,8 +629,15 @@ public void testDistCpWithIterator() throws Exception {
GenericTestUtils
.createFiles(remoteFS, source, getDepth(), getWidth(), getWidth());
+ GenericTestUtils.LogCapturer log =
Review comment:
Thanx @steveloughran, do you suggest that we should have two options
like -useiteratorforsource and -useiteratorfortarget. Do you think in that case
we would be able to save out on memory? since the target list is being build as
part of CopyCommitter, so even if one takes the normal path, We would get OOM,
just `when` will differ?
Regarding the log stuff, That was the only thing I could think of, to
confirm if iterator was used. And during migration to Log4J2, Will moving to
something like this will also be of no help:
https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java#L285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 581098)
Time Spent: 1h 10m (was: 1h)
> DistCp: Use Iterator for listing target directory as well
> ---------------------------------------------------------
>
> Key: HADOOP-17620
> URL: https://issues.apache.org/jira/browse/HADOOP-17620
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.3.1, 3.4.0
> Reporter: Ayush Saxena
> Assignee: Ayush Saxena
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Use iterator for listing target directory as well, when {{-useiterator}}
> option is specified.
> Target is listed when delete option is specified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
