Hello Balazs Hevele, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24553
to look at the new patch set (#14).
Change subject: WIP IMPALA-15163: Add geometry bounding box predicate pushdown
for Parquet
......................................................................
WIP IMPALA-15163: Add geometry bounding box predicate pushdown for Parquet
Implement Parquet row group skipping based on GEOMETRY column
bounding box statistics. This commit stays entirely within Parquet
metadata: writing the bounding box to Iceberg manifests is left to a
follow-up commit.
Writer side: the Parquet table writer tracks the envelope of all
geometries written to each column chunk and stores it in
ColumnMetaData.geospatial_statistics.bbox, holding xmin/xmax/ymin/ymax
as doubles (see parquet-format Geospatial.md). The generic Statistics
min/max are left unset, as geometry has no sort order and the spec
leaves them undefined for it; only null_count is written there.
The Iceberg lower/upper bounds of a GEOMETRY column stay unset, as
before this commit.
The box has to cover every value in the column chunk, or a reader
pruning on it would drop matching rows, so collection fails safe: if
any non-NULL geometry cannot be bounded, no box is written for that
row group at all. ComputeWkbBBox() therefore reports EMPTY and
MALFORMED separately instead of returning one ambiguous sentinel, as
the two need opposite handling - an empty geometry has no extent and is
simply skipped, while an unparseable value abandons the box. This also
covers values Impala can store but not bound, since ParseWkbHeader()
accepts only the six 2D WKB types: the Z/M/ZM variants (st_pointz()
writes WKB type 1001), GEOMETRYCOLLECTION and EWKB. Bounding those
rather than skipping them is left to a follow-up; IMPALA-15168 tracks
support for non-2D geometries.
Reader side: HdfsScanNode.tryComputeGeoBBoxPredicate() recognizes
st_envintersects(CONST, geom_col) conjuncts, computes the constant
geometry's envelope via ESRI, and emits TGeoBBoxPredicate to the
backend. EvaluateGeoBBoxPredicates() in the Parquet scanner checks
bbox overlap against the row group's geospatial statistics and
increments NumStatsFilteredRowGroups on miss.
Iceberg does not support predicate pushdown for geometry types yet
(see https://github.com/apache/iceberg/pull/14101), so the filtering
has to happen at the Parquet level. Note that pruning during planning
would be still possible in Impala once the bounds are written to
Iceberg's DataFile metadata returned by planFiles() - but this is not
implemented in this commit either.
The geospatial expression rewrite rules (NormalizeGeospatialRelations,
PointEnvIntersects, AddEnvIntersects) are now enabled for every
geospatial library except NONE, instead of only for HIVE_ESRI. Without
AddEnvIntersectsRule no st_envintersects is generated for relations
such as st_intersects, so nothing could be pushed down in the default
WKB_EXPERIMENTAL mode. This also uncovered a precondition in
PointEnvIntersectsRule that only accepted BINARY constants and would
have failed on GEOMETRY ones.
Testing:
- create table with Impala and verify that bbox is written and used
- check relations that do pushdown (st_envintersects, etc..) and
st_disjoint that must not do pushdown
- verify that a row group holding only empty/NULL geometries does
not get a bounding box
- verify that a row group holding a value that cannot be bounded (a Z
point) gets no bounding box at all, and that the row is still
returned by a query whose window covers only that value
- unit tests for the EMPTY / MALFORMED / OK outcomes of
ComputeWkbBBox()
- no interop tests yet, would need to find an engine that writes
statistics
Assisted-by: Claude Opus 5 (Claude Code)
Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exprs/geo/wkb-serialization-test.cc
M be/src/exprs/geo/wkb-serialization.cc
M be/src/exprs/geo/wkb-serialization.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/rewrite/PointEnvIntersectsRule.java
M
java/hive-geospatial-functions/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry-bbox.test
M tests/query_test/test_geospatial_functions.py
13 files changed, 668 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/24553/14
--
To view, visit http://gerrit.cloudera.org:8080/24553
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
Gerrit-Change-Number: 24553
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
