huan233usc opened a new pull request, #17451:
URL: https://github.com/apache/iceberg/pull/17451
Follow-up to #17333.
#17333 collects the average serialized WKB size in `FieldMetrics`, but
`ParquetMetrics` previously discarded it while assembling `Metrics`. As a
result, the value could not reach a `DataFile` or be exposed to v4
`content_stats` adapters.
This change keeps the metric in Iceberg's shared in-memory file model
without changing the v1-v3 manifest schema.
- Carry per-field average non-null value sizes through `Metrics`,
`ContentFile`, file builders, copies, and filtering.
- Populate the map from Parquet field metrics and verify `DataWriter` to
`DataFile` propagation.
- Expose v4 content stats through the legacy `ContentFile` view used by
manifest readers.
- Preserve compatibility with older Java-serialized `Metrics` objects.
- Align the v4 spec so `avg_value_size_in_bytes` includes `geometry` and
`geography`.
## Scope and dependencies
- This is stacked on #17333; the first two commits come from that PR. The
follow-up change is commit `018c80224`.
- v1-v3 manifest schemas remain unchanged. In particular, v3 manifests do
not persist this optional metric.
- The v4 write-direction content-stats wrapper is being added in #16936.
This PR supplies the `ContentFile` carrier/accessor that wrapper needs and the
v4 read bridge; #16936 can consume `ContentFile.avgValueSizes()` after rebasing.
## Test Plan
- [x] API serialization, including an old serialized `Metrics` fixture
- [x] Core content-stats map and tracked-file adapter tests
- [x] Parquet metrics and `DataWriter` Geo propagation tests
- [x] Generic and Spark Geo value-size writer tests from #17333
- [x] API, Core, and Parquet RevAPI checks
- [x] API, Core, and Parquet Spotless checks
## Verification Commands
```bash
./gradlew :iceberg-api:test --tests
org.apache.iceberg.TestMetricsSerialization \
:iceberg-core:test --tests org.apache.iceberg.TestContentStatsBackedMap \
--tests org.apache.iceberg.TestTrackedFileAdapters \
:iceberg-data:test \
--tests
org.apache.iceberg.parquet.TestParquetMetrics.testMetricsForGeospatialTypes \
:iceberg-parquet:test \
--tests
org.apache.iceberg.parquet.TestParquetDataWriter.testGeospatialRoundTrip
./gradlew :iceberg-core:test \
--tests org.apache.iceberg.TestValueSizeFieldMetrics \
--tests org.apache.iceberg.TestFieldStatsStruct \
:iceberg-parquet:test --tests
org.apache.iceberg.parquet.TestParquetValueWriters \
:iceberg-spark:iceberg-spark-4.1_2.13:test \
--tests
org.apache.iceberg.spark.data.TestSparkParquetWriter.testGeospatialAvgValueSizeMetrics
./gradlew :iceberg-api:revapi :iceberg-core:revapi :iceberg-parquet:revapi \
:iceberg-api:spotlessCheck :iceberg-core:spotlessCheck \
:iceberg-parquet:spotlessCheck
```
---
**AI Disclosure**
- Model: GPT-5
- Platform/Tool: Codex
- Human Oversight: partially reviewed
- Prompt Summary: Propagate Geo average WKB sizes through Metrics and
ContentFile for v4 content stats in a separate follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
