Hello Balazs Hevele, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24553
to look at the new patch set (#11).
Change subject: POC: IMPALA-15163 Add geometry bounding box predicate pushdown
for Parquet
......................................................................
POC: IMPALA-15163 Add geometry bounding box predicate pushdown for Parquet
Implement Parquet row group skipping based on GEOMETRY column
bounding box statistics. Iceberg does not support predicate
pushdown for geometry types yet (see
https://github.com/apache/iceberg/pull/14101), so this operates
at the Parquet level, using the geospatial statistics written to
the column chunks.
Note that pruning during planning would be still possible in Impala -
bounds are available in Iceberg's DataFile metadata returned by
planFiles() - but this is not implemented in this commit.
Writer side: the Parquet table writer tracks the envelope of all
geometries written to each column chunk. The bounding box is then
written to the two metadata systems in the encoding each of them
defines:
- Parquet: ColumnMetaData.geospatial_statistics.bbox, holding
xmin/xmax/ymin/ymax as doubles (see parquet-format
Geospatial.md). The generic Statistics min/max are left unset,
as geometry has no sort order and the spec leaves them undefined
for it; only null_count is written there.
- Iceberg manifests (via the normal commit path): the lower and
upper bounds are the two corners of the box, serialized like
Iceberg's GeospatialBound, i.e. 16 bytes each holding
xmin||ymin and xmax||ymax as LE doubles.
Empty and malformed geometries have no extent, so they do not
contribute to the box, and do not cause one to be written either.
Reader side: HdfsScanNode.tryComputeGeoBBoxPredicate() recognizes
st_envintersects(CONST, geom_col) conjuncts, computes the constant
geometry's envelope via ESRI, and emits TGeoBBoxPredicate to the
backend. EvaluateGeoBBoxPredicates() in the Parquet scanner checks
bbox overlap against the row group's geospatial statistics (not
Iceberg metadata) and increments NumStatsFilteredRowGroups on miss.
Also fixes Iceberg metadata table queries (e.g. data_files) for
tables with GEOMETRY columns: excludes 'readable_metrics' from the
scan projection because Iceberg 1.11.0's Conversions.fromByteBuffer
does not handle GEOMETRY (throws UnsupportedOperationException).
The column and its nested fields return NULL until a future Iceberg
release adds support.
Testing:
- create table with Impala and verify that bbox is written and used
- check relations that do pushdown (st_envintersects, etc..) and
st_disjoint that must not do pushdown
- use metadata table queries to verify bbox stats are written to
Iceberg manifests
- verify that a row group holding only empty/NULL geometries does
not get a bounding box
- no interop tests yet, would need to find an engine that writes
statistics
Assisted-by: Claude Opus 5 (Claude Code)
Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/rewrite/PointEnvIntersectsRule.java
M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
M
java/hive-geospatial-functions/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry-bbox.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test
M tests/query_test/test_geospatial_functions.py
13 files changed, 704 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/24553/11
--
To view, visit http://gerrit.cloudera.org:8080/24553
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
Gerrit-Change-Number: 24553
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
