Hi everyone, The next Parquet footer sync, Session 5, is tomorrow, September 1, at 10am PT / 1pm ET / 7pm CET.
To join the invite, please subscribe to the Parquet calendar: https://calendar.google.com/calendar/u/0?cid=OTU2YjU0NGRlMGM5MjEwYTZlYzg1OWNhMGNjYTViMTZjNjhlYmRkNmJjYmY0ZmRlNGIzZjFjZTE5MzUwZGU2NkBncm91cC5jYWxlbmRhci5nb29nbGUuY29t Video call link: https://meet.google.com/zem-shuo-ubt We have already converged on several high-level design principles for the new footer, including a modular structure and column-major organization. In this session, we will focus on one of the main remaining open questions: how the data inside those modules should be encoded. The current options span a spectrum from simple Thrift-based representations to more specialized encodings that improve compactness and random access. The main challenge is balancing format simplicity, size, decode efficiency, and random-access capability. We will review the options and their tradeoffs and discuss which properties we want to prioritize in the final design. A more detailed description and comparison of the options can be found here: https://docs.google.com/document/d/1eiygLyg_jcEiPnF_F6XI01dmX2DDa6UjDJH9ZAJCKe8/edit?tab=t.rcgvwgqebv46 Looking forward to your thoughts and discussion. Best, Jiayi
