nssalian commented on code in PR #17557:
URL: https://github.com/apache/iceberg/pull/17557#discussion_r3972737303
##########
parquet/src/test/java/org/apache/iceberg/parquet/TestVariantMetrics.java:
##########
@@ -466,6 +473,122 @@ public void testShreddedStringBoundsAcrossRowGroups()
throws IOException {
.isEqualTo(Variants.of(supplementary));
}
+ @Test
+ public void testMissingNullCountAcrossRowGroups() throws IOException {
+ // A variant column chunk may omit null_count in its footer statistics,
which Parquet reports
+ // as -1. When one row group is missing the count and another has it, the
total must be
+ // reported as unknown rather than summing the -1 into a lower count.
+ ParquetMetadata footer =
+ footer(Variant.of(EMPTY, Variants.of(1)), null, null); // 1 value, 2
nulls
+
+ // build a two row group footer: the first as written, the second with
null_count removed
+ // from the variant sub columns
+ BlockMetaData withCount = footer.getBlocks().get(0);
+ BlockMetaData withoutCount =
dropVariantNullCounts(footer.getBlocks().get(0));
+ ParquetMetadata twoRowGroups =
+ new ParquetMetadata(footer.getFileMetaData(),
Lists.newArrayList(withCount, withoutCount));
+
+ Metrics metrics =
+ ParquetUtil.footerMetrics(twoRowGroups, Stream.empty(),
MetricsConfig.getDefault());
+
+ // the variant column (id 2) null count is unknown because one row group
did not report it
+ assertThat(metrics.nullValueCounts()).doesNotContainKey(2);
+ assertThat(metrics.valueCounts()).containsEntry(2, 6L);
+ }
+
+ @Test
+ public void testShreddedNullVariantsWithMissingNullCount() throws
IOException {
+ // For a shredded column where the values are either shredded into
typed_value or are null
Review Comment:
I think cleaning up to something like this:
```
// A shredded column's value column holds only null variants, so its null
count comes from the
// value count, not the footer. Shredded bounds survive a row group that
omits null_count.
```
would suffice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
