Mihaly Szjatinya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24986
Change subject: IMPALA-15004: [WIP] Write NDV stats to Iceberg Puffin files in COMPUTE STATS ...................................................................... IMPALA-15004: [WIP] Write NDV stats to Iceberg Puffin files in COMPUTE STATS COMPUTE STATS on an Iceberg table previously stored column NDV statistics only in the Hive Metastore (HMS). This change additionally writes them as an Iceberg Puffin statistics file using the standard apache-datasketches-theta-v1 blob type, so the NDVs live with the table metadata and interoperate with other engines such as Trino and Spark. Impala's existing Puffin read path already consumes such files, so the written stats are picked up on the next metadata load. For an Iceberg table, COMPUTE STATS now issues a third child query that computes a compact theta sketch per column with ds_theta_sketch_iceberg, which encodes values per Iceberg Appendix D single-value serialization. The coordinator forwards the serialized sketches to catalogd in TAlterTableUpdateStatsParams; catalogd writes them to a Puffin file with the Iceberg Java API and commits it via UpdateStatistics against the snapshot the stats were computed for. Testing: - tests/query_test/test_iceberg.py:: TestIcebergTableWithPuffinStats.test_compute_stats_writes_puffin: read->write Puffin cycle within Impala - tests/custom_cluster/test_iceberg_trino_puffin_sketches.py: Impala<->Trino interop over Puffin theta sketches Regression: - tests/metadata/test_compute_stats.py - tests/query_test/test_iceberg.py::TestIcebergTable::test_compute_stats - tests/query_test/test_iceberg.py::TestIcebergTableWithPuffinStats - tests/query_test/test_datasketches.py::TestDatasketches::test_theta Change-Id: I0112492a9fb156b5317d8e0a6bb3dcf18ae6eeec Assisted-by: Claude Opus 4.8 (GitHub Copilot) --- M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/service/IcebergPuffinStatsWriter.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-puffin-stats.test A tests/custom_cluster/test_iceberg_trino_puffin_sketches.py M tests/query_test/test_iceberg.py 10 files changed, 705 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24986/1 -- To view, visit http://gerrit.cloudera.org:8080/24986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0112492a9fb156b5317d8e0a6bb3dcf18ae6eeec Gerrit-Change-Number: 24986 Gerrit-PatchSet: 1 Gerrit-Owner: Mihaly Szjatinya <[email protected]>
