Francois Saint-Jacques created ARROW-8884: ---------------------------------------------
Summary: [C++] Listing files with S3FileSystem is slow Key: ARROW-8884 URL: https://issues.apache.org/jira/browse/ARROW-8884 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Francois Saint-Jacques Listing files on S3 is slow due to the recursive nature of the algorithm. The following change modifies the behavior of the S3Result to include all objects but no "grouping" (directories). This lower dramatically the number of HTTP calls. {code:c++} diff --git a/cpp/src/arrow/filesystem/s3fs.cc b/cpp/src/arrow/filesystem/s3fs.cc index 70c87f46ec..98a40b17a2 100644 --- a/cpp/src/arrow/filesystem/s3fs.cc +++ b/cpp/src/arrow/filesystem/s3fs.cc @@ -986,7 +986,7 @@ class S3FileSystem::Impl { if (!prefix.empty()) { req.SetPrefix(ToAwsString(prefix) + kSep); } - req.SetDelimiter(Aws::String() + kSep); + // req.SetDelimiter(Aws::String() + kSep); req.SetMaxKeys(kListObjectsMaxKeys); while (true) { {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)