timwaizenegger commented on issue #388: URL: https://github.com/apache/arrow-rs-object-store/issues/388#issuecomment-2962777906
> This is incorrect because offset listing doesn't require the results to be ordered; it only requires the filesystem to have stable results, meaning that the order remains consistent within the filesystem itself. I though so too initially. But the offset filter is just that; a filter. It doesn't "skip until offset reached", it just filters based on lexicographical comparisons. The "skip until" couldn't even work without sorted results. What if the token file gets deleted? Now you can't know how far to skip any more. How about we add sorted output for `LocalFileSystem` as an opt-in feature; as mentioned here? https://github.com/apache/arrow-rs-object-store/pull/389#discussion_r2115996113 No harm in doing that, right? Can document the pros and cons and it would enable this paged-listing use case for local files. I'm happy to propose changes if you can give me a few pointers. Could be a cargo "feature" since you probably decide at compile time if you need this and not at runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
