Jiayi-Wang-db opened a new pull request, #611:
URL: https://github.com/apache/parquet-format/pull/611
### Rationale for this change
The traditional array-of-structs footer requires readers to walk metadata
for every column even
when a query projects only a few columns. The modular footer transposes
metadata into independent
column-major modules, but storing every encoded array as a Thrift `binary`
field obscures the
boundary between the Thrift control structures and the actual array encoding.
This draft explores an explicit metadata-page model analogous to Parquet
data pages: a
Compact-Thrift header followed by a raw, uncompressed array payload. It
keeps placement,
statistics, and page indexes as independent modules and supports projected
random access without
coupling placement to optional statistics.
### What changes are included in this PR?
* Add the base modular-footer data model and its README entry.
* Add a detailed array-page proposal covering framing, indexing,
compatibility, validation, and
evaluation criteria.
* Add normative Thrift definitions for metadata-page headers and directories.
* Define `BIT_PACKED` for dense arrays and `SPARSE` for sparse optional
values.
* Define a two-level directory so page-index metadata does not expand the
always-read root.
* Keep metadata-page payloads uncompressed and outside Thrift fields.
### Do these changes have PoC implementations?
Not yet. This is a draft specification intended to establish the wire model
before updating the
existing modular-footer PoC. The proposal lists the required size,
projection, sparse-statistics,
malformed-input, and cross-implementation evaluations.
<!-- Closes #530 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
