jordepic commented on code in PR #2647:
URL: https://github.com/apache/iceberg-rust/pull/2647#discussion_r3650479758
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -619,7 +621,113 @@ impl RecordBatchTransformer {
})
.collect()
}
+}
+
+/// Look up an Iceberg field id from an Arrow field's `PARQUET:field_id`
metadata.
+fn arrow_field_id(field: &Field) -> Option<i32> {
+ field
+ .metadata()
+ .get(PARQUET_FIELD_ID_META_KEY)
+ .and_then(|s| s.parse::<i32>().ok())
+}
+
+/// Mirrors iceberg-java logic, resolve by field ID instead of position.
+fn cast_schema_to_target(array: &ArrayRef, target_type: &DataType) ->
Result<ArrayRef> {
+ match target_type {
+ DataType::Struct(target_children) => {
+ let source = array.as_struct_opt().ok_or_else(|| {
+ Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "expected a struct array to promote to
{target_type:?}, got {:?}",
+ array.data_type()
+ ),
+ )
+ })?;
+ let mut source_by_id: HashMap<i32, usize> = HashMap::new();
Review Comment:
Restructured along exactly these lines. `ColumnSource::Promote` now carries
a `PromotePlan` built once in `generate_transform_operations`
(`PromotePlan::build`), which resolves all nested field-id matching, the target
`Fields`/`FieldRef`s, and absent children up front; `plan.apply` per batch is
just index lookups and array assembly — no `HashMap`, no metadata parsing. The
`Fields`/`FieldRef` clones now live in the plan, so the per-batch clones are
Arc bumps only.
On the REE sub-point: a nested absent child has to match the declared arrow
type of the target struct child (`StructArray::try_new` validates children
against its `Fields`), and the REE representation the top-level `Add` path uses
would change that type. So absent children keep `new_null_array`, which per
batch is a null-buffer allocation rather than a materialized value vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
