xanderbailey commented on code in PR #2249:
URL: https://github.com/apache/iceberg-rust/pull/2249#discussion_r3059478893
##########
crates/iceberg/src/writer/file_writer/parquet_writer.rs:
##########
@@ -67,12 +70,33 @@ impl ParquetWriterBuilder {
schema: SchemaRef,
match_mode: FieldMatchMode,
) -> Self {
+ let props = Self::add_iceberg_schema_metadata(props, &schema);
Self {
props,
schema,
match_mode,
}
}
+
+ /// Adds the `iceberg.schema` key-value metadata to the Parquet writer
properties.
+ ///
+ /// This embeds the full Iceberg schema JSON (including field IDs, types,
and
+ /// schema ID) into the Parquet file footer.
+ fn add_iceberg_schema_metadata(props: WriterProperties, schema: &Schema)
-> WriterProperties {
+ let schema_json = serde_json::to_string(schema)
+ .expect("Iceberg schema serialization to JSON should not fail");
+
+ let iceberg_kv = KeyValue::new(ICEBERG_SCHEMA_KEY.to_string(),
schema_json);
+
+ // Preserve any existing key-value metadata from the caller
+ let mut kv_metadata =
props.key_value_metadata().cloned().unwrap_or_default();
+ kv_metadata.push(iceberg_kv);
Review Comment:
If a caller has already set `ICEBERG_SCHEMA_KEY` then we'll add it twice
here. Java uses a map so I think it's not an issue there but I feel like we're
better off reporting an error in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
