[
https://issues.apache.org/jira/browse/HADOOP-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649610#action_12649610
]
Amar Kamat commented on HADOOP-4623:
------------------------------------
bq. That means each operation involves link manipulation and objection creation.
There is no object creation involved. Node is passed by the jobtracker which it
maintains per tracker/node. TIP is already created by the job as part of
job-initialization. Only data structure updates will happen using object
references.
bq. Use a HashMap mapping nodes to fix sized arrays of tips. The fix size
should be the number of slots per node.
Wouldnt this result into memory wastage as there might be many other jobs
running in parallel and the whole(expanded) _running-cache_ might not be used.
How will you take care of holes in the array?
----
I think I should benchmark the patch again and see if the results are
consistent. If yes then we can go ahead with the patch and file an optimization
jira for running-cache. Thoughts?
> Running tasks are not maintained by JobInProgress if speculation is off
> -----------------------------------------------------------------------
>
> Key: HADOOP-4623
> URL: https://issues.apache.org/jira/browse/HADOOP-4623
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4623-v1.1.patch, HADOOP-4623-v1.2.patch
>
>
> {{JobInProgress}} doesnt maintain any structure for running tasks if
> speculation is turned _off_. {{getRunningMapCache()}} in {{JobInProgress}}
> exposes the running map cache. This api returns an empty {{Map}} if
> speculation turned off.
> _Usage_ :
> {{CapicityScheduler}} requires a list of running tasks for both speculated
> and non-speculated jobs. See HADOOP-4558 to see how this issue affects
> {{CapacityScheduler}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
