GitHub user Xuanwo edited a discussion: Concurrent support for opendal list
The issue of adding concurrent support for opendal list has been discussed for a long time: https://github.com/apache/opendal/issues/987. Listing a large prefix on S3 is a very slow process, as demonstrated in this image.  It would be a great improvement for OpenDAL to support concurrent list. The challenge is how to offer this as a user-friendly and effective API. --- My current idea is that we can introduce a `partitions` option for list. The partitions must not overlap to ensure that the list returns only unique results. The usage will be like: ```rust let result = op .list_with("dir/") .partitions(["a", "m"]) .await?; ``` opendal will start three list underhood: `list("dir/")`, `list("dir/a")`, `list("dir/m")`. Users can speed up the process by adding `concurrent` for them: ```rust let result = op .list_with("dir/") .partitions(["a", "m"]) .concurrent(3) .await?; ``` After adding `partitions`, the return results is not in order anymore. --- What do you think? GitHub link: https://github.com/apache/opendal/discussions/6115 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
