This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 900ec3ee38 docs: Revise Parquet metadata module documentation (#10796)
900ec3ee38 is described below
commit 900ec3ee38276ab651e210c4a85b38f8a8a61bcf
Author: Ed Seidl <[email protected]>
AuthorDate: Tue Aug 25 12:45:33 2026 -0700
docs: Revise Parquet metadata module documentation (#10796)
# Which issue does this PR close?
N/A
# Rationale for this change
The form of the Parquet metadata has changed a great deal since the
module documentation was written. It's time for a small refresh.
# What changes are included in this PR?
Mention the new `PageIndex` structure and remove the ascii art diagram
(sorry @alamb) that is now out of date. Since we no longer have two
parallel sets of metadata objects, the section on the interplay between
representation is no longer needed.
# Are these changes tested?
N/A
# Are there any user-facing changes?
No, just documentation
---
parquet/src/file/metadata/mod.rs | 43 +++-------------------------------------
1 file changed, 3 insertions(+), 40 deletions(-)
diff --git a/parquet/src/file/metadata/mod.rs b/parquet/src/file/metadata/mod.rs
index e7f7199177..4eeed39088 100644
--- a/parquet/src/file/metadata/mod.rs
+++ b/parquet/src/file/metadata/mod.rs
@@ -32,10 +32,12 @@
//! within a Row Group including encoding and compression information,
//! number of values, statistics, etc.
//!
+//! * [`PageIndex`]: Metadata and statistics used to reduce page-level I/O.
+//!
//! # APIs for working with Parquet Metadata
//!
//! The Parquet readers and writers in this crate handle reading and writing
-//! metadata into parquet files. To work with metadata directly,
+//! metadata into Parquet files. To work with metadata directly,
//! the following APIs are available:
//!
//! * [`ParquetMetaDataReader`] for reading metadata from an I/O source (sync
and async)
@@ -47,45 +49,6 @@
//! Please see [`external_metadata.rs`]
//!
//! [`external_metadata.rs`]:
https://github.com/apache/arrow-rs/tree/master/parquet/examples/external_metadata.rs
-//!
-//! # Metadata Encodings and Structures
-//!
-//! There are three different encodings of Parquet Metadata in this crate:
-//!
-//! 1. `bytes`:encoded with the Thrift `TCompactProtocol` as defined in
-//! [parquet.thrift]
-//!
-//! 2. [`format`]: Rust structures automatically generated by the thrift
compiler
-//! from [parquet.thrift]. These structures are low level and mirror
-//! the thrift definitions.
-//!
-//! 3. [`file::metadata`] (this module): Easier to use Rust structures
-//! with a more idiomatic API. Note that, confusingly, some but not all
-//! of these structures have the same name as the [`format`] structures.
-//!
-//! [`file::metadata`]: crate::file::metadata
-//! [parquet.thrift]:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
-//!
-//! Graphically, this is how the different structures relate to each other:
-//!
-//! ```text
-//! ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ─ ─
─ ─ ─ ─ ─
-//! ┌──────────────┐ │
┌───────────────────────┐ │
-//! │ │ ColumnIndex │ ││
ParquetMetaData │
-//! └──────────────┘ │
└───────────────────────┘ │
-//! ┌──────────────┐ │ ┌────────────────┐
│┌───────────────────────┐
-//! │ ..0x24.. │ ◀────▶ │ OffsetIndex │ │ ◀────▶ │
ParquetMetaData │ │
-//! └──────────────┘ │ └────────────────┘
│└───────────────────────┘
-//! ... │ ...
│
-//! │ ┌──────────────────┐ │
┌──────────────────┐
-//! bytes │ FileMetaData* │ │ │
FileMetaData* │ │
-//! (thrift encoded) │ └──────────────────┘ │
└──────────────────┘
-//! ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ ─ ─ ─ ─ ─ ─ ─ ─
─ ─ ─ ─ ─ ┘
-//!
-//! format::meta structures file::metadata
structures
-//!
-//! * Same name, different struct
-//! ```
mod footer_tail;
mod memory;
mod options;
