Christopher Boumalhab created SPARK-54138:
---------------------------------------------
Summary: Enforce constant configuration parameter for Hll and
Theta Sketches Aggregation
Key: SPARK-54138
URL: https://issues.apache.org/jira/browse/SPARK-54138
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Christopher Boumalhab
*Description:*
Add validation to ensure that the sketch configuration parameter (e.g.,
{{{}lgConfigK/lgNomEntries{}}}) in both {{HllSketchAgg}} and {{ThetaSketchAgg}}
is constant. If the configuration expression ({{{}right{}}}) is not foldable,
throw a {{QueryExecutionErrors.*MustBeConstantError(prettyName)}} to prevent
runtime issues caused by dynamic or non-deterministic configurations.
*Rationale:*
The configuration parameter defines key precision and memory characteristics of
HLL and Theta sketches. It must be known at analysis time to guarantee
deterministic aggregation behavior and consistent accuracy. Allowing
non-constant parameters could lead to unpredictable results and incorrect
aggregations.
*Acceptance Criteria:*
* Queries fail with a clear error if the configuration parameter
({{{}right{}}}) is not foldable.
* Constant parameters are accepted and validated with existing checks (e.g.,
{{{}checkLgK{}}}).
* Existing valid queries with constant configuration values continue to work
without changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
