Mihaly Szjatinya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24982
Change subject: IMPALA-15459: Add Iceberg-encoded theta sketch flavor for Puffin NDV stats ...................................................................... IMPALA-15459: Add Iceberg-encoded theta sketch flavor for Puffin NDV stats COMPUTE STATS will write column NDV statistics as Iceberg Puffin apache-datasketches-theta-v1 blobs (IMPALA-15004). For cross-engine interop (Trino, Spark) each value must be hashed into the sketch using Iceberg Appendix D single-value serialization (int=4B, float=4B, date=4B, boolean=1B). Impala's ds_theta_sketch() hashes values via the DataSketches typed update() overloads, which widen int/date/boolean to 8 bytes and float to double. This matches Hive's datasketches UDF but not the Iceberg Puffin spec, so INT/FLOAT/DATE/BOOLEAN sketches would not interoperate with Trino/Spark. Changing ds_theta_sketch() directly would break byte-level interop with Hive-produced sketches (datasketches-theta.test). This patch adds an internal-only aggregate flavor, ds_theta_sketch_iceberg, that encodes values with Iceberg Appendix D serialization. COMPUTE STATS (IMPALA-15004) uses it when writing Puffin NDV blobs; the user-facing ds_theta_sketch() keeps its Hive-compatible encoding. - BE: added AggregateFunctions::DsThetaUpdateIceberg<T> with Appendix D specialisations for BOOLEAN (1 byte), INT (4-byte LE), FLOAT (4-byte LE) and DATE (4-byte LE). The other Iceberg column types (BIGINT, DOUBLE, STRING, TIMESTAMP, DECIMAL) reuse DsThetaUpdate, which already encodes identically. - FE: registered the internal ds_theta_sketch_iceberg aggregate via a new DS_THETA_ICEBERG_UPDATE_SYMBOL map in BuiltinsDb.java. TINYINT/SMALLINT are intentionally omitted: Iceberg has no such column types, and a stray narrow value would implicitly widen to INT and get the correct encoding. Testing: - datasketches-theta.test: added a section that sketches the same values as INT and BIGINT with both flavors and compares them with ds_theta_exclude; Regression: - datasketches-theta.test - aggregate-functions-test.cc DsThetaSketch.*: Change-Id: I78c76c38dffa9807eb13ad465afab0b80f31984a Assisted-by: Claude Opus 4.8 (GitHub Copilot) --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test 4 files changed, 133 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/24982/1 -- To view, visit http://gerrit.cloudera.org:8080/24982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I78c76c38dffa9807eb13ad465afab0b80f31984a Gerrit-Change-Number: 24982 Gerrit-PatchSet: 1 Gerrit-Owner: Mihaly Szjatinya <[email protected]>
