pvary commented on code in PR #17320:
URL: https://github.com/apache/iceberg/pull/17320#discussion_r3657307334
##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -280,29 +280,91 @@ public ParquetValueReader<?> struct(
}
}
- int constantDefinitionLevel = type.getMaxDefinitionLevel(currentPath());
+ String[] structPath = currentPath();
+ int constantDefinitionLevel = type.getMaxDefinitionLevel(structPath);
List<Types.NestedField> expectedFields = expected.fields();
+
+ // When a nullable struct's projected fields are all constants (initial
defaults, metadata
+ // columns, or partition values) with no real column read from the file,
the struct has no
+ // per-row definition level to indicate whether an ancestor struct is
null. In that case,
+ // borrow the definition level from a real leaf column that exists under
the struct in the
+ // file so a null ancestor is not incorrectly materialized as a struct
of default values.
+ boolean hasRealFieldReader =
+ expectedFields.stream().anyMatch(field ->
readersById.containsKey(field.fieldId()));
+ ColumnDescriptor probe = null;
+ Integer probeHostId = null;
+ if (!hasRealFieldReader && constantDefinitionLevel > 0) {
+ probe = firstLeafUnder(structPath);
+ if (probe != null) {
+ probeHostId = probeHostFieldId(expectedFields);
+ }
+ }
+
List<ParquetValueReader<?>> reorderedFields =
Lists.newArrayListWithExpectedSize(expectedFields.size());
-
for (Types.NestedField field : expectedFields) {
int id = field.fieldId();
ParquetValueReader<?> reader =
ParquetValueReaders.replaceWithMetadataReader(
id, readersById.get(id), idToConstant,
constantDefinitionLevel);
- reorderedFields.add(defaultReader(field, reader,
constantDefinitionLevel));
+ boolean hostsProbe = probeHostId != null && id == probeHostId;
+ reorderedFields.add(
Review Comment:
Yeah, that's a good point.
Also checked that if a Spark query results in a struct, and it contains the
metadata columns, then it is converted to straight column reads.
So, yeah, you are correct. We are fine with this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
