[
https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689382#comment-15689382
]
Thomas Demoor commented on HADOOP-13826:
----------------------------------------
Mmmh, sorry, I caused this when introducing the bounded pool to protect
against OOM when streaming the upload from a memory buffer. To my defense, at
time of implementation we were programming against aws-sdk 1.7.4 where the docs
don't seem to mention this :(
The bounded threadpool blocks the clients (data producers) so memory usage is
bounded. The TransferManager has built in throttling so they do not need this
and advocate using an unbounded pool. However, the TransferManager does not
offer functionality to do "block-based" uploads. So for S3ABlockOutputStream we
need a different solution. The first thing that came to my mind was two
threadpools (1 unbounded for TransferManager, 1 bounded for Block) which have a
shared limit for number of active (uploading) threads but would like something
simpler if possible. Have not yet studied [~mackrorysd]'s solution in detail,
will have a look.
> S3A Deadlock in multipart copy due to thread pool limits.
> ---------------------------------------------------------
>
> Key: HADOOP-13826
> URL: https://issues.apache.org/jira/browse/HADOOP-13826
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Sean Mackrory
> Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The
> TransferManager javadocs
> (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
> explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread
> pool with a bounded work queue as control tasks may submit subtasks that
> can't complete until all sub tasks complete. Using an incorrectly configured
> thread pool may cause a deadlock (I.E. the work queue is filled with control
> tasks that can't finish until subtasks complete but subtasks can't execute
> because the queue is filled).{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
