tustvold commented on issue #388: URL: https://github.com/apache/arrow-rs-object-store/issues/388#issuecomment-2962819678
> What if the token file gets deleted? Now you can't know how far to skip any more. The start offset is a filter, so it would just yield the values after that point. The same "issue" would occur if you inserted a new file before or after the iteration position. FWIW no object stores provide stable pagination AFAIK. > it would enable this paged-listing use case for local files. But it doesn't, it just fudges around it by fetching everything from the filesystem, sorting it and then discarding those less than a given filter. Perhaps it might be worth taking a step back and explaining what your use-case and we can then better advise on how to achieve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
