[ https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312778#comment-15312778 ]
Daniel Weeks edited comment on HADOOP-12878 at 6/2/16 5:56 PM: --------------------------------------------------------------- Actually, looking at the ResourceRequest, is it possible to use 'ANY' = '*', which looks like it would be intended for this purpose. https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java#L131 was (Author: dweeks): Actually, looking at the ResourceRequest, is it possible to use 'ANY' (*), which looks like it would be intended for this purpose. https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java#L131 > Impersonate hosts in s3a for better data locality handling > ---------------------------------------------------------- > > Key: HADOOP-12878 > URL: https://issues.apache.org/jira/browse/HADOOP-12878 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.8.0 > Reporter: Thomas Demoor > Assignee: Thomas Demoor > > Currently, {{localhost}} is passed as locality for each block, causing all > blocks involved in job to initially target the same node (RM), before being > moved by the scheduler (to a rack-local node). This reduces parallelism for > jobs (with short-lived mappers). > We should mimic Azures implementation: a config setting > {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list > of hostnames in the cluster to return to {{getFileBlockLocations}}. > Possible optimization: for larger systems, it might be better to return N > (5?) random hostnames to prevent passing a huge array (the downstream code > assumes size = O(3)). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org