[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731695#comment-14731695
]
Aaron Fabbri commented on HADOOP-11684:
---------------------------------------
I guess the argument for CallerRunsPolicy is:
1. One line(ish) change. Low risk for regressions.
2. Eventual, possible OOM better than immediate exception fail.
I'm still pretty new to codebase. Anyone have example where many threads would
be entering S3AFilesystem to do writes?
I still intend to spend more time reviewing the patch to convince myself it is
correct.. just didn't get to it this week. A unit test for the new blocking
executor would be sweet as well.
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
