Xi Liang created SPARK-61819:
--------------------------------

             Summary: Do not normalize conditional distinct counts with 
non-deterministic conditions, and read the config once
                 Key: SPARK-61819
                 URL: https://issues.apache.org/jira/browse/SPARK-61819
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Xi Liang


SPARK-56898 (fix version 4.4.0, not released yet) made 
{{RewriteDistinctAggregates}} normalize {{COUNT(DISTINCT IF(cond, base, 
NULL))}} and {{COUNT(DISTINCT CASE WHEN cond THEN base END)}} to 
{{COUNT(DISTINCT base) FILTER (WHERE cond)}} while 
{{spark.sql.optimizer.rewriteCountDistinctConditional.enabled}} is true, which 
is its default. Several such counts over one base then share one group of the 
{{Expand}} that the rule builds. The rule also guards that group's distinct 
key, the value that the {{Expand}} projects for it, with the union of the 
counts' filters: {{IF(cond_1 OR ... OR cond_n, base, NULL)}} (the 
union-of-filters guard). Rows that match no filter then project NULL.

For each value of the GROUP BY keys, the first aggregate above the {{Expand}} 
then outputs, for that group, one row per distinct base value among the rows 
that match some condition, plus at most one NULL row. Without the 
normalization, it outputs no fewer: for each condition, one row per distinct 
base value among the rows that match that condition, plus at most one NULL row. 
Without the guard, the distinct key is the base itself on every row, so the 
aggregate outputs, for that group, one row per distinct base value of all rows. 
Results stay correct, because the FILTER clauses still select the rows of each 
count, but that output can be much larger.

The rule can still normalize counts and then build their group without the 
guard, in two cases:
# *A non-deterministic condition.* The guard applies only when every filter of 
the group is deterministic, because it evaluates the filters a second time, in 
the distinct key. The normalization does not check determinism, so a count 
whose condition is non-deterministic is normalized and joins the group of the 
other counts over its base, and then no count of that group is guarded. 
Analysis rejects non-deterministic expressions in aggregate arguments, and 
{{CollapseProject}} inlines only deterministic expressions, so we know of no 
query that reaches this case; the rule should not depend on that.
# *The config changing while the rule runs*, for example when another thread 
sets it on the same session. For one {{Aggregate}}, the config is read twice: 
when the counts are normalized, and when the guard is decided. If it is turned 
off in between, the normalized counts are kept and their guard is skipped.

Proposed change:
* Do not normalize a count whose condition is non-deterministic. It keeps its 
own group, whose distinct key is its original {{IF}} or {{CASE WHEN}}; the 
other counts over the same base can still share a guarded group.
* Read the config once for each {{Aggregate}} that the rule rewrites, and use 
that value for both the normalization and the guard.
* Add tests for both cases, for the existing check that only a count with one 
argument is normalized, for the number of rows that the first aggregate above 
the {{Expand}} outputs, and for a deterministic condition over a column that a 
subquery computes with {{input_file_name()}}.
* Set the config to true in the subexpression elimination benchmark of 
{{ExpandBenchmark}}, which compares two settings of subexpression elimination 
on the normalized plan, so that it measures that plan whatever the default. The 
file's two other benchmarks of this rewrite already set the config in each of 
their cases.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to