JingsongLi commented on code in PR #1080:
URL: https://github.com/apache/paimon-rust/pull/1080#discussion_r4226011531
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -1351,7 +1378,8 @@ fn read_type_projection_mask(
let mut full = path.clone();
full.extend(parts);
full
- }),
+ })
+ .filter(|full| !null_residuals.contains(full)),
Review Comment:
[P1] Preserve structural leaves when skipping null residuals
An all-null residual can still carry definition levels that distinguish an
existing empty object from a missing object. This filter can remove every leaf
beneath a requested group, causing Parquet to omit the entire group from the
projected schema.
I reproduced this with shredding schema `v: ROW<obj: ROW<x: VARIANT>, y:
BIGINT>` and rows `{"obj":{},"y":2}` and `{"y":3}`, extracting `$.obj` as
STRING and `$.y` as DOUBLE. Both `obj.value` and `obj.typed_value.x.value` are
null throughout the file, so this filter removes the whole `obj` group. The
first row then returns NULL for `$.obj` instead of `"{}"`. Reading the same
file with the previous projection preserves the empty object and matches the
unshredded extraction.
The same issue can leave only root metadata when extracting an entirely
absent residual-only VARIANT child, which produces `Variant.value column must
be BinaryArray` instead of null results.
Please retain enough selected leaves to preserve the requested groups'
definition/repetition levels, and cover these residual-only child and
empty-object cases in regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
