adriangb commented on PR #11031: URL: https://github.com/apache/arrow-rs/pull/11031#issuecomment-5808767333
It looks like we at least don't use the footer-only path. The arrow-rs example and the original caching use case both load the page index before writing, and as far as I can tell no one else requested it, and [[#6464](https://github.com/apache/arrow-rs/issues/6464)](https://github.com/apache/arrow-rs/issues/6464) reports the leftover offsets as a bug. I suggest that we keep the current behavior (always clear the locations that the writer did not write), add the `breaking change` label, and close #6464. Then we document two points on `ParquetMetaDataWriter`: to include the page index, load it before you write; to cache the footer only, store the raw footer bytes, which already contain the original offsets. This is simpler than a boolean, and the output is always self-consistent. If someone needs the old behavior later, we can add the option then. Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
