huan233usc opened a new pull request, #17297:
URL: https://github.com/apache/iceberg/pull/17297
## Summary
Spark 4.1 currently falls back to the row reader whenever a projection
contains a geometry or geography column, even when Parquet vectorization is
enabled. This adds vectorized Geo reads without introducing a GeoArrow
representation: Iceberg continues to store pure WKB, Arrow carries it in
variable-width binary vectors, and the Spark column-vector boundary
reconstructs `GeometryVal`/`GeographyVal`. Geometry values receive the SRID
derived from the Iceberg schema CRS.
- Map Iceberg geometry and geography primitives to Arrow binary fields and
read their Parquet logical annotations through the existing `VARBINARY` batch
path.
- Restore typed Spark Geo values at `IcebergArrowColumnVector`, including
non-default geometry SRIDs.
- Allow `SparkBatch` to select columnar Parquet reads for top-level Geo
primitives.
- Cover dictionary-enabled and plain pages, null values, WKB round trips,
EPSG:3857 geometry, CRS84 geography, and existing DML/deletion-vector scans.
Nested structs remain on the existing row-reader fallback because nested
vectorized reads are outside this change. The Parquet representation remains
byte-identical WKB, and this does not add spatial parsing or bbox behavior to
the reader.
## Test Plan
- [x] Arrow schema conversion test for geometry and geography
- [x] Direct Spark batched-reader test with dictionary enabled and disabled
- [x] Full Spark Geo SQL/DML suite with vectorization on and off
- [x] Arrow and Spark 4.1 Spotless checks
## Verification Commands
```bash
./gradlew :iceberg-arrow:test --tests
org.apache.iceberg.arrow.TestArrowSchemaUtil
./gradlew :iceberg-spark:iceberg-spark-4.1_2.13:test \
--tests
org.apache.iceberg.spark.data.vectorized.parquet.TestParquetVectorizedReads.testGeospatialTypes
\
--tests org.apache.iceberg.spark.sql.TestSparkGeospatial
./gradlew :iceberg-arrow:spotlessCheck \
:iceberg-spark:iceberg-spark-4.1_2.13:spotlessCheck
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
