[
https://issues.apache.org/jira/browse/HADOOP-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582920#action_12582920
]
Amar Kamat commented on HADOOP-3104:
------------------------------------
Here are some comments
1) The javadoc comments should not mention the default value. That might change
and will require code change too. So you can keep the earlier comment as is and
just add the comment about the wait parameter.
2) I think mapred.map.multithreadedrunner.backoff seems more appropriate than
mapred.map.multithreadedrunner.waitwhennothreads, comments?
3) 10ms seems too short. I was wondering what if we double it everytime.
Something like 10,20,40,80 ...
> MultithreadMapRunner keeps consuming records even if trheads are not available
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3104
> URL: https://issues.apache.org/jira/browse/HADOOP-3104
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.1
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 0.16.2
>
> Attachments: patch3104.txt
>
>
> The ExecutorService execute() method does not block when there are not
> threads available, it queues up the runnables until there are threads.
> The problem is that all key/values are read and kept in memory for the task,
> with large datasets this will create a OOM exception.
> Have to figure out how to use the execute in blocking fashion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
