bharath-techie commented on PR #11297: URL: https://github.com/apache/arrow-rs/pull/11297#issuecomment-5911104452
> @bharath-techie is this something like the solution you mentioned in Yes this is exactly what we have done - so basically the context is - in OpenSearch scoped page index cache implementation, we have 3 caches, one is footer metadata that has file metadata and row groups - so for a file we'll have this entire thing cached. We have offset index cache - scoped for row groups and columns We have column index cache - scoped for row groups and columns So eventually when datafusion implements granular metadata cache replacing the current full file metadata - one way is to take the pieces from cache and stitch it together. So we might need this PR in such case, WDYT ? not sure if there are other ways to stitch and give the metadata per query without clone. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
