Hi all. I am sharing as a separate thread the proposal for the footer change we have been working on: https://docs.google.com/document/d/1kZS_DM_J8n6NKff3vDQPD1Y4xyDdRceYFANUE0bOfb0/edit .
The proposal outlines the technical aspects of the design and the experimental results of shadow testing this in production workloads. I would like to discuss the proposal's most salient points in the next sync: 1. the use of flatbuffers as footer serialization format 2. the additional limitations imposed on parquet files (row group size limit, row group max num row limit) I would prefer comments on the google doc to facilitate async discussion. Thank you,
