westonpace commented on issue #33759: URL: https://github.com/apache/arrow/issues/33759#issuecomment-1405367881
Thanks for the explanation. I do think we cache the metadata per-file when opening a dataset. The original thought was that a user might open a dataset and then scan it multiple times. If we cache the data the first time then we can save time on future reads. However, if you have large metadata and many files then I think that becomes problematic. I will open a separate issue for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
