amogh-jahagirdar commented on code in PR #17320:
URL: https://github.com/apache/iceberg/pull/17320#discussion_r3653138489
##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -280,29 +280,91 @@ public ParquetValueReader<?> struct(
}
}
- int constantDefinitionLevel = type.getMaxDefinitionLevel(currentPath());
+ String[] structPath = currentPath();
+ int constantDefinitionLevel = type.getMaxDefinitionLevel(structPath);
List<Types.NestedField> expectedFields = expected.fields();
+
+ // When a nullable struct's projected fields are all constants (initial
defaults, metadata
+ // columns, or partition values) with no real column read from the file,
the struct has no
+ // per-row definition level to indicate whether an ancestor struct is
null. In that case,
+ // borrow the definition level from a real leaf column that exists under
the struct in the
+ // file so a null ancestor is not incorrectly materialized as a struct
of default values.
+ boolean hasRealFieldReader =
+ expectedFields.stream().anyMatch(field ->
readersById.containsKey(field.fieldId()));
+ ColumnDescriptor probe = null;
+ Integer probeHostId = null;
+ if (!hasRealFieldReader && constantDefinitionLevel > 0) {
+ probe = firstLeafUnder(structPath);
+ if (probe != null) {
+ probeHostId = probeHostFieldId(expectedFields);
+ }
+ }
+
List<ParquetValueReader<?>> reorderedFields =
Lists.newArrayListWithExpectedSize(expectedFields.size());
-
for (Types.NestedField field : expectedFields) {
int id = field.fieldId();
ParquetValueReader<?> reader =
ParquetValueReaders.replaceWithMetadataReader(
id, readersById.get(id), idToConstant,
constantDefinitionLevel);
- reorderedFields.add(defaultReader(field, reader,
constantDefinitionLevel));
+ boolean hostsProbe = probeHostId != null && id == probeHostId;
+ reorderedFields.add(
Review Comment:
Constants beyond default values also already had this issue, it's not a new
regression in this change. But I'll see if it's easy to handle that case as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
