Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24969
Change subject: IMPALA-XXXXX: Write geometry bounding box statistics to Iceberg ...................................................................... IMPALA-XXXXX: Write geometry bounding box statistics to Iceberg Follow-up to IMPALA-15163, which added the geospatial bounding box to the Parquet column chunk metadata. This commit writes it to the Iceberg manifests too, so that other engines reading the table's metadata can use it, and so that Impala can prune files during planning once Iceberg supports geometry predicates. The box goes into the data file's lower and upper bounds, which for a geometry column are its two corners, serialized like Iceberg's GeospatialBound: 16 bytes each, holding xmin||ymin and xmax||ymax as LE doubles (see https://iceberg.apache.org/spec/#bound-serialization). The bounds follow the same rule as the Parquet bounding box and are written only when it is, so a data file whose geometries have no extent, or that holds a value which could not be bounded, gets no geometry bounds. This breaks Iceberg metadata table queries (e.g. data_files) for tables with GEOMETRY columns: constructing 'readable_metrics' calls Conversions.fromByteBuffer(), which throws UnsupportedOperationException for GEOMETRY in Iceberg 1.11.0. As a workaround IcebergMetadataScanner excludes 'readable_metrics' from the scan projection for such tables, so the column and its nested fields return NULL instead of failing the query. Iceberg already has a fix for this (https://github.com/apache/iceberg/pull/16607), so the workaround can be removed once we get Iceberg 1.12.0. As 'readable_metrics' is excluded from the scan, it never reaches complex type serialization, so it is also exempted from the rule that rejects GEOMETRY nested in a complex type in the select list of a top-level query (IMPALA-15244). Real, materializable nested GEOMETRY is still rejected. Testing: - use metadata table queries to verify that bbox stats are written to Iceberg manifests - verify that a data file holding a value that cannot be bounded gets no geometry lower/upper bounds - verify that readable_metrics and its nested fields read as NULL for a table with a GEOMETRY column, instead of failing the query - no interop tests yet, would need to find an engine that reads geometry bounds from the manifests Assisted-by: Claude Opus 5 (Claude Code) Change-Id: Id230a05c3a9e75f0c4f25381a31f15aaa63ecd91 --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry-bbox.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test 5 files changed, 157 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/24969/1 -- To view, visit http://gerrit.cloudera.org:8080/24969 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id230a05c3a9e75f0c4f25381a31f15aaa63ecd91 Gerrit-Change-Number: 24969 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
