Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24559 )
Change subject: IMPALA-12700: Take Iceberg Parquet Bloom filter properties into account ...................................................................... IMPALA-12700: Take Iceberg Parquet Bloom filter properties into account When writing Parquet files for Iceberg tables, Impala previously only honored its own Bloom filter mechanisms: the 'parquet_bloom_filter_write' query option and the 'parquet.bloom.filter.columns' table property. It ignored the Iceberg-native Bloom filter write properties, so a table configured for Bloom filters by another engine (Spark, Flink, PyIceberg) got no Bloom filters when written by Impala. This change makes Impala honor the Iceberg-native properties for Iceberg tables: - write.parquet.bloom-filter-enabled.column.<col> - write.parquet.bloom-filter-max-bytes (default 1 MiB) - write.parquet.bloom-filter-ndv.column.<col> - write.parquet.bloom-filter-fpp.column.<col> For each enabled column the Bloom filter bitset size is derived the same way parquet-java sizes it: from the expected number of distinct values (NDV) and the false positive probability (FPP), capped by 'bloom-filter-max-bytes'. The NDV is taken from the per-column property if set, otherwise from the column's computed NDV statistic (COMPUTE STATS) when available, otherwise the filter is sized at max-bytes (matching parquet-java when the NDV is unknown). Using the column's stats as a fallback is an improvement over parquet-java, which has no access to such statistics. The implementation is frontend-only: the backend Parquet writer already consumes a column -> bitset-bytes map, so the size is computed in the FE. The Iceberg-derived map is merged with the Impala-specific 'parquet.bloom.filter.columns' map; the Impala-specific property takes precedence for any column present in both, keeping existing behavior backward compatible. The property values are validated at CREATE TABLE / ALTER TABLE SET TBLPROPERTIES time (a positive 32-bit max-bytes, a positive NDV, and an FPP in (0, 1)), mirroring the existing Iceberg row-group-size / page-size validation. Another engine can set these properties without going through Impala's DDL validation, so invalid values are also detected when Impala writes the table. Such writes are not failed: the invalid properties are ignored (no Bloom filters are produced, as before) and a warning is surfaced to the user during analysis of the write. This covers every statement that emits Iceberg data files - INSERT, UPDATE, MERGE and OPTIMIZE - which are the paths where these properties would otherwise take effect. Testing: * New FE unit test IcebergBloomFilterUtilTest covering optimal sizing, property validation, the NDV-from-stats fallback and column-name handling. * New FE analysis tests in AnalyzeDDLTest (TestIcebergParquetBloomFilter Properties) for invalid property values and setting the properties on a non-parquet Iceberg table, covering both CREATE TABLE and ALTER TABLE SET TBLPROPERTIES. * New e2e tests in test_parquet_bloom_filter.py that create Iceberg tables with the properties, write data and verify the Bloom filters, plus a test that injects an invalid property via Hive and asserts the warning is surfaced (and the write still succeeds) for INSERT, UPDATE, MERGE, and OPTIMIZE. Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e Assisted-by: Claude Opus 5 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24559 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergMergeImpl.java M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/planner/ParquetBloomFilterTblPropParserTest.java A fe/src/test/java/org/apache/impala/util/IcebergBloomFilterUtilTest.java M tests/query_test/test_parquet_bloom_filter.py 15 files changed, 944 insertions(+), 76 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e Gerrit-Change-Number: 24559 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
