flaneur2020 commented on issue #274: URL: https://github.com/apache/arrow-rs-object-store/issues/274#issuecomment-3064796173
[SlateDB](https://github.com/slatedb/slatedb) currently uses a transparent [object_store wrapper](https://github.com/slatedb/slatedb/blob/main/slatedb/src/cached_object_store/object_store.rs) for chunked splitting and caching (similar to @crepererum's approach, it also contains a `head()` before `get()` to fetch size). however, we're facing some challenges with DST (having to mock the filesystem for the cache storage), and it would be great to leverage community solutions (or contribute to upstream) for chunking/caching rather than maintaining our own implementation in the longer term. iiuc, there are two paths proposed in this issue: - **object_store wrapper**: as described by @crepererum, it still leverages the object_store trait and has lower integration costs for the existing codebases. - **a `GetClient` built on top of object_store**: as described by @tustvold, it contains a more sophisticated chunking api design both would greatly benefit projects like SlateDB by providing standardized, upstream-supported mechanisms to reuse. could you share which approach is currently in progress? we'd also be happy to contribute to this. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
