sundapeng commented on code in PR #1080:
URL: https://github.com/apache/paimon-rust/pull/1080#discussion_r4227929900
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -1351,7 +1378,8 @@ fn read_type_projection_mask(
let mut full = path.clone();
full.extend(parts);
full
- }),
+ })
+ .filter(|full| !null_residuals.contains(full)),
Review Comment:
Good catch, thank you. Both cases share one root cause: a shredded field
group is required, and whether the field exists is carried by the definition
levels of its `typed_value` subtree, not by the field's own `value` leaf — its
null statistics say nothing about that structure.
`skippable_residuals` now enforces that per read: a projected all-null
residual `typed_value.<key>.value` is left unread only when a selected leaf
survives under the field's `typed_value` subtree. Candidates are decided
deepest-first, so a surviving nested residual also preserves its ancestors'
groups. With that:
- your `{"obj":{}}` case keeps `obj.typed_value.x.value` selected, so
`$.obj` reads as `"{}"`;
- a residual-only VARIANT child keeps its sole leaf, so it no longer
disappears from the projected schema and returns nulls instead of the
`Variant.value column must be BinaryArray` error.
Both are covered as regression tests
(`shredded_variant_extraction_preserves_null_groups`,
`shredded_variant_extraction_preserves_absent_variant_child`), plus
`null_residual_skip_keeps_field_structure` pinning the rule directly. The
plain-typed layout that motivated this PR keeps the full skip: the field's
`typed_value` typed leaf survives, so its all-null residual stays unread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
