[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939139#comment-14939139
]
Aaron Fabbri commented on HADOOP-11684:
---------------------------------------
Ok. With that explanation I'm fine with the new defaults. Worst case a config
change can be made to re-tune.
I'm a little confused about the JVM out of memory (OOM) issue: Either way, you
have many threads in play. Whether they block, or start uploading themselves.
Is there something about uploading that dramatically increases the memory usage
per thread?
Certainly, the blocking executor is more robust at limiting the number of
concurrent uploads. I like it, it just takes more effort for me to understand
it is bug-free.
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch,
> HADOOP-11684-003.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
