[
https://issues.apache.org/jira/browse/HADOOP-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated HADOOP-1719:
-------------------------------
Status: Patch Available (was: Open)
Resubmitting the same patch for the current trunk. I ran some benchmarks with
this patch and found these results.
|Without patch |3693 ms|4172 ms|57m43s|
|With patch |3500 ms|3193 ms|56m10s|
These are some runs for 100,500 nodes. What I saw in the logs is as follows
* In the current trunk, this is the sequence of copy events immediately
followed by atleast one copy-done event.
* With this patch I saw that some times the reducer waits as it runs out of
copier threads and waits for the copy to finish and free the copiers. So all
the *num-copier* thread get busy fetching the map outputs.
> Improve the utilization of shuffle copier threads
> -------------------------------------------------
>
> Key: HADOOP-1719
> URL: https://issues.apache.org/jira/browse/HADOOP-1719
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Attachments: 1719.1.patch, 1719.patch, HADOOP-1719.patch
>
>
> In the current design, the scheduling of copies is done and the scheduler
> (the main loop in fetchOutputs) won't schedule anything until it hears back
> from at least one of the copier threads. Due to this, the main loop won't
> query the TaskTracker asking for new map locations and may not be using all
> the copiers effectively. This may not be an issue for small-sized map
> outputs, where at steady state, the frequency of such notifications is
> frequent.
> Ideally, we should schedule all what we can, and, depending on how busy we
> currently are, query the tasktracker for more map locations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
