[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Labels: (was: ShuffleKeepalive) Hadoop Flags: Reviewed Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0-v8.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Attachment: MAPREDUCE-5787-2.4.0-v8.patch Okay the latest patch looks good. But I did one more minor round of micro-optimization to avoid reconstructing the output path again and again. Will commit it once jenkins says okay. Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0-v8.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2 and branch-2.4. Thanks Rajesh! Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0-v8.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Attachment: MAPREDUCE-5787-2.4.0-v6.patch MAPREDUCE-5787-2.4.0-v5-v6-diff.patch Your patch has some non-standard formatting and I had time to kill. Here are two files attached - one which shows my additional changes (formatting, refactoring and javadoc) and one with all the changes. I think there is one more big issue to take care of. Because we are avoiding hitting the disk for fetching the ShuffleHader multiple times, you are keeping them in memory. There is a small bump in the memory foot-print after this patch. Asssuming 500 reducers fetching outputs of 500 maps all at the same time, this translates to 250KShuffleHeaders and assuming 40bytes per header, the ShuffleHandler needs 10MB extra memory now. Which isn't significant, our index-cache size itself is 10MB by default, but the problem is this increased memory-foot-print is unbounded. Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v7.patch Addressed Vinod's concern on increased in memory due to mapOutputFileName and IndexRecord. It is possible to configure the cache via mapreduce.shuffle.mapoutput-info.meta.cache.size. (default value is 1000). String locaDirAllocator computations will be carried out twice if number of mapId goes past this limit. Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0-v7.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v5.patch Incorporated review comments from Vinod Can we also change the MapReduce fetcher to use keep-alive depending on whether it is enabled or not? - HttpURLConnection will automatically use persistent connection when keep-alive and Content-Length headers are properly set. So, there is no need to change the fetcher code. Suggestion for Configuration renames - Fixed Add both to the mapred-default.xml - Fixed LOG KeepAliveParam along with other things like jobId, mapId etc. -Fixed populateHeaders. We are already parsing jobID, ApplicationId etc as part of sendMapOutput. We should avoid doing the string parsing multiple times. Is setting CONTENT_LENGTH important? Even so, for doing it, we are reading the index-record two times - Yes, content-length is very much needed for this. Fixed multiple parsing issue. Instead of re-defining new constants like CONNECTION_HEADER in ShuffleHandler, can you use the standard constants in java (HttpHeaders)? - Fixed Finally, can you reuse code between the two tests? - Fixed Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5787: --- Fix Version/s: (was: 2.4.0) Target Version/s: 2.4.0 Status: Open (was: Patch Available) Looked at the patch, some comments: Can we also change the MapReduce fetcher to use keep-alive depending on whether it is enabled or not? Suggestion for Configuration renames - mapreduce.shuffle.enable.keep.alive - mapreduce.shuffle.connection-keep-alive.enable - mapreduce.shuffle.enable.keep.alive.timeout - mapreduce.shuffle.connection-keep-alive.timeout - Add both to the mapred-default.xml ShuffleHandler.java - SHUFFLE_KEEP_ALIVE - SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED - DEFAULT_SHUFFLE_KEEP_ALIVE - DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED - SHUFFLE_KEEP_ALIVE_TIME_OUT - SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT - DEFAULT_SHUFFLE_KEEP_ALIVE_TIME_OUT - DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT - Variable keepAlive - connectionKeepAliveEnabled - Similarly keepAliveTimeOut - connectionKeepAliveTimeOut - LOG KeepAliveParam along with other things like jobId, mapId etc. - populateHeaders -- We are already parsing jobID, ApplicationId etc as part of sendMapOutput. We should avoid doing the string parsing multiple times. -- Is setting CONTENT_LENGTH important? Even so, for doing it, we are reading the index-record two times - once here and once while sending the output. This will have performance impact. - Instead of re-defining new constants like CONNECTION_HEADER in ShuffleHandler, can you use the standard constants in java (HttpHeaders)? Finally, can you reuse code between the two tests? There seems to be quite a bit of duplication. Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: (was: BUG-14568-v3-branch-2.4.0.patch) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v3.patch Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: BUG-14568-v3-branch-2.4.0.patch Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: (was: MAPREDUCE-5787-2.4.0-v3.patch) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v3.patch Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Open (was: Patch Available) Will incorporate the review comments from Gopal (https://reviews.apache.org/r/19264/diff/1/?file=521462#file521462line585) and upload the patch Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v3.patch Incorporated review comments from Gopal (https://reviews.apache.org/r/19264/diff/1/?file=521462#file521462line585) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated MAPREDUCE-5787: --- Priority: Critical (was: Major) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated MAPREDUCE-5787: --- Affects Version/s: 2.4.0 Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated MAPREDUCE-5787: --- Component/s: nodemanager Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated MAPREDUCE-5787: --- Labels: ShuffleKeepalive (was: ) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated MAPREDUCE-5787: --- Fix Version/s: 2.4.0 Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: (was: MAPREDUCE-5787-2.4.0-v3.patch) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v4.patch Renaming the patch as v4. Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Open (was: Patch Available) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: nodemanager Affects Versions: 2.4.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: ShuffleKeepalive Fix For: 2.4.0 Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v2.patch - Keep-Alive is disabled by default. - Keep-Alive can be enabled by setting mapreduce.shuffle.enable.keep.alive - Timeout parameter can be adjusted using mapreduce.shuffle.enable.keep.alive.timeout - There is add-on facility wherein Keep-Alive can be enabled via request URL by adding keepAlive=true parameter. This will allow frameworks like Tez to benefit from Keep-Alive connections without affecting any MR jobs (for which the keep-alive connections are disabled by default). Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0-v2.patch, MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0.patch Modify ShuffleHandler to support Keep-Alive --- Key: MAPREDUCE-5787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: MAPREDUCE-5787-2.4.0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
