[
https://issues.apache.org/jira/browse/YARN-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333483#comment-17333483
]
Eric Badger commented on YARN-7713:
-----------------------------------
Thanks for taking this up, [~ChrisKarampeazis]. I noticed that you weren't a
contributor in JIRA yet so I've added you as one. You may now assign JIRAs to
yourself in all of the Hadoop projects (YARN, Common, HDFS, Mapreduce).
In general I think the PR looks good, but I think it would be nice and not too
awfully difficult to sort the list of files to be localized by file size and
then split the list into chunks based on that. That way we don't end up with 1
thread downloading 4 files of 2 KB and another thread downloading 4 files of 4
GB.
> Add parallel copying of directories into FSDownload
> ---------------------------------------------------
>
> Key: YARN-7713
> URL: https://issues.apache.org/jira/browse/YARN-7713
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Miklos Szegedi
> Assignee: Christos Karampeazis-Papadakis
> Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> YARN currently copies directories sequentially when localizing. This could be
> improved to do in parallel, since the source blocks are normally on different
> nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
