[
https://issues.apache.org/jira/browse/TEZ-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989939#comment-13989939
]
Siddharth Seth commented on TEZ-1102:
-------------------------------------
Rajesh, took a quick look. Looks good, what the patch is doing for connection
establishment and validation. Merging the Fetchers, ShuffleManager etc can be
done later.
Couple of comments
- Instead of passing in the Configuration to HttpConnection - it'd be better to
pass in the parameters itself, such as keepAlive, timeouts etc. Otherwise this
ends up reading from conf for each connection - which is very avoidable.
- Can the entier URL be constructed outside and passed in - using the new
method added in ShuffleUtils. That should remove unnecessary bits like
partition, appId which are otherwise not required for a connection.
The fetchers now have a bunch of unused variables, which can be removed.
> Abstract out connection management logic in shuffle code
> --------------------------------------------------------
>
> Key: TEZ-1102
> URL: https://issues.apache.org/jira/browse/TEZ-1102
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1102-v1.patch
>
>
> Unordered / ordered shuffle manage their own connections. Currently
> enhancements / bug fixes on connection management has to be done in both
> places.
> Purpose of this JIRA is abstract out the connection management in
> unordered/ordered shuffle code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
