[
https://issues.apache.org/jira/browse/ARROW-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou resolved ARROW-10788.
------------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 8818
[https://github.com/apache/arrow/pull/8818]
> [C++] Make S3 recursive walks parallel
> --------------------------------------
>
> Key: ARROW-10788
> URL: https://issues.apache.org/jira/browse/ARROW-10788
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Antoine Pitrou
> Assignee: Antoine Pitrou
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Doing a recursive S3 directory walk using GetFileInfo(Selector) currently
> lists all encountered directories serially, waiting for the results of one
> directory listing (or portion thereof) before launching the next one.
> Instead, we should use the Async APIs provided by the AWS SDK to parallelize
> HTTP requests as much as possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
