[
https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384339#comment-14384339
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5036:
----------------------------------------------------
I think changing it to an ephemeral port is good in general.
But the recent feature of NM-recovery (YARN-1336) can make this a problem.
- ShuffleHandler starts on port A
- MR AM launches a container, gets port A as the target location for the
intermediate output, passes this to the reducer
- NM restarts and ShuffleHandler starts on port B now.
- Reducer fails to perform shuffle.
Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to
send changed aux-service port info to all apps when a node restarts. /cc
[~jlowe], [~djp]
> Default shuffle handler port should not be 8080
> -----------------------------------------------
>
> Key: MAPREDUCE-5036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.0.3-alpha
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Fix For: 2.4.0
>
> Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch,
> MAPREDUCE-5036.patch
>
>
> The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is
> a pretty common port for web services, and is likely to cause unnecessary
> port conflicts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
