[
https://issues.apache.org/jira/browse/SPARK-61819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61819:
-----------------------------------
Labels: pull-request-available (was: )
> Do not normalize conditional distinct counts with non-deterministic
> conditions, and read the config once
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61819
> URL: https://issues.apache.org/jira/browse/SPARK-61819
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Xi Liang
> Priority: Major
> Labels: pull-request-available
>
> SPARK-56898 (fix version 4.4.0, not released yet) made
> {{RewriteDistinctAggregates}} normalize {{COUNT(DISTINCT IF(cond, base,
> NULL))}} and {{COUNT(DISTINCT CASE WHEN cond THEN base END)}} to
> {{COUNT(DISTINCT base) FILTER (WHERE cond)}} while
> {{spark.sql.optimizer.rewriteCountDistinctConditional.enabled}} is true,
> which is its default. Several such counts over one base then share one group
> of the {{Expand}} that the rule builds. The rule also guards that group's
> distinct key, the value that the {{Expand}} projects for it, with the union
> of the counts' filters: {{IF(cond_1 OR ... OR cond_n, base, NULL)}} (the
> union-of-filters guard). Rows that match no filter then project NULL.
> For each value of the GROUP BY keys, the first aggregate above the {{Expand}}
> then outputs, for that group, one row per distinct base value among the rows
> that match some condition, plus at most one NULL row. Without the
> normalization, it outputs no fewer: for each condition, one row per distinct
> base value among the rows that match that condition, plus at most one NULL
> row. Without the guard, the distinct key is the base itself on every row, so
> the aggregate outputs, for that group, one row per distinct base value of all
> rows. Results stay correct, because the FILTER clauses still select the rows
> of each count, but that output can be much larger.
> The rule can still normalize counts and then build their group without the
> guard, in two cases:
> # *A non-deterministic condition.* The guard applies only when every filter
> of the group is deterministic, because it evaluates the filters a second
> time, in the distinct key. The normalization does not check determinism, so a
> count whose condition is non-deterministic is normalized and joins the group
> of the other counts over its base, and then no count of that group is
> guarded. Analysis rejects non-deterministic expressions in aggregate
> arguments, and {{CollapseProject}} inlines only deterministic expressions, so
> we know of no query that reaches this case; the rule should not depend on
> that.
> # *The config changing while the rule runs*, for example when another thread
> sets it on the same session. For one {{Aggregate}}, the config is read twice:
> when the counts are normalized, and when the guard is decided. If it is
> turned off in between, the normalized counts are kept and their guard is
> skipped.
> Proposed change:
> * Do not normalize a count whose condition is non-deterministic. It keeps its
> own group, whose distinct key is its original {{IF}} or {{CASE WHEN}}; the
> other counts over the same base can still share a guarded group.
> * Read the config once for each {{Aggregate}} that the rule rewrites, and use
> that value for both the normalization and the guard.
> * Add tests for both cases, for the existing check that only a count with one
> argument is normalized, for the number of rows that the first aggregate above
> the {{Expand}} outputs, and for a deterministic condition over a column that
> a subquery computes with {{input_file_name()}}.
> * Set the config to true in the subexpression elimination benchmark of
> {{ExpandBenchmark}}, which compares two settings of subexpression elimination
> on the normalized plan, so that it measures that plan whatever the default.
> The file's two other benchmarks of this rewrite already set the config in
> each of their cases.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]