alamb commented on code in PR #11297:
URL: https://github.com/apache/arrow-rs/pull/11297#discussion_r4298848982
##########
parquet/src/file/metadata/thrift/mod.rs:
##########
@@ -757,6 +757,14 @@ pub(crate) fn parquet_metadata_from_bytes(
buf: &[u8],
options: Option<&ParquetMetaDataOptions>,
) -> Result<ParquetMetaData> {
+ let (file_metadata, row_groups) =
file_metadata_and_row_groups_from_bytes(buf, options)?;
+ Ok(ParquetMetaData::new(file_metadata, row_groups))
+}
+
+pub(crate) fn file_metadata_and_row_groups_from_bytes(
Review Comment:
this makes sense
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -116,15 +116,18 @@ pub(crate) use writer::ThriftMetadataWriter;
/// This structure is read by the various readers in this crate or can be read
/// directly from a file using the [`ParquetMetaDataReader`] struct.
///
+/// The individual fields of this structure are stored in a way that make
cloning
+/// this structure low-cost.
+///
/// See the [`ParquetMetaDataBuilder`] to create and modify this structure.
///
/// [`parquet.thrift`]:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
#[derive(Debug, Clone)]
pub struct ParquetMetaData {
/// File level metadata
- file_metadata: FileMetaData,
+ file_metadata: Arc<FileMetaData>,
/// Row group metadata
- row_groups: Vec<RowGroupMetaData>,
+ row_groups: Arc<Vec<RowGroupMetaData>>,
Review Comment:
I think we can make this `Arc<Vec<Arc<RowGroupMetaData>>>` in the future if
that turns out to be a problem
Another minor thing here to save a few bytes (maybe 16 bytes per instance)
is to instead of
```rust
row_groups: Arc<Vec<RowGroupMetaData>>,
```
remove the Vec part:
```rust
row_groups: Arc<[RowGroupMetaData]>,
```
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -347,14 +346,17 @@ impl ParquetMetaDataBuilder {
}
/// Adds a row group to the metadata
+ ///
+ /// If the row groups are shared with another `ParquetMetaData`, this
clones
Review Comment:
💯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
