[
https://issues.apache.org/jira/browse/ARROW-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246706#comment-17246706
]
Weston Pace commented on ARROW-10846:
-------------------------------------
By default AWS spins up a new thread per task [1][2]. Their reasoning comes
down to "a modern OS is the best thread pool"[3]. I'm not sure I entirely
agree with this but I can see the reasoning. It would be interesting to
experiment with at some point. It has led AWS developers to exhaust the OS
thread limit if they are not careful [4][5]. AWS exposes an executor interface
that can be supplied if different behavior is desired.
I agree though that it is better to let the S3 filesystem (and other
async-aware filesystems) call their async implementation directly instead of
wrapping their sync implementation. However, to keep the filesystem API that
must be implemented simpler, do we want to provide default implementations for
these async APIs so this can be an optional thing?
> [C++] Add async filesystem operations
> -------------------------------------
>
> Key: ARROW-10846
> URL: https://issues.apache.org/jira/browse/ARROW-10846
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Antoine Pitrou
> Priority: Major
>
> It would probably be useful to have Future-returning variants of some
> filesystem operations (at least {{GetFileInfo}} and
> {{OpenInput(File|Stream)}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
