[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906766#comment-14906766
]
Aaron Fabbri commented on HADOOP-11684:
---------------------------------------
Looks pretty good. A couple of questions:
1. Can you comment on how you arrived at the new default values for
`fs.s3a.threads.max`, `fs.s3a.threads.core`, and `fs.s3a.max.total.tasks`?
(i.e. we no longer need artificially large thread pools to hack around the fact
that the old implementation would throw an exception when `threads *
fs.s3a.max.total.tasks` requests in flight was exceeded).
2. Since the behavior of these parameters has changed, should we add an entry
to release notes notifying folks of the new behavior?
Would you like some help testing?
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch,
> HADOOP-11684-003.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
