GitHub user Xuanwo added a comment to the discussion: Concurrent support for
opendal list
> I check the s3 list objects v1/v2 APIs, which return common prefixes.
> Thinking aloud with the API. What if users are listing:
>
> ```rust
> let result = op
> .list_with("di")
> .partitions(["r", "m"])
> .concurrent(3)
> .await?;
> ```
>
> Use s3 as an example, we will request:
>
> * `di`
> * `dir`
> * `dim`
In fact, we will list the following range:
- `di..dir`
- `dir..dim`
- `dim..`
Underhood, we will have something like the following three concurrent list:
- `op.lister("di").take_while(|v|future::ready(v.path < "dir"))`
- `op.lister("dir").take_while(|v|future::ready(v.path < "dim"))`
- `op.lister("dim")
So, opendal will guarantee that the lister returns non-overlapping results
after partitioning.
Maybe we can add some validation here or just sort input partitions?
GitHub link:
https://github.com/apache/opendal/discussions/6115#discussioncomment-13088984
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
