[
https://issues.apache.org/jira/browse/SPARK-60934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60934:
-----------------------------------
Labels: pull-request-available (was: )
> [SQL] Treat seedless TABLESAMPLE and aes_encrypt without an IV as
> non-deterministic
> -----------------------------------------------------------------------------------
>
> Key: SPARK-60934
> URL: https://issues.apache.org/jira/browse/SPARK-60934
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: James Xu
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> Two operators draw fresh randomness at execution but report {{deterministic =
> true}}. Rules that trust the flag then compute the operator once and reuse
> the result, so two references change from two independent draws to one shared
> draw.
> *Scenario 1: seedless TABLESAMPLE.*
> {{TABLESAMPLE}} without {{REPEATABLE}} parses to {{Sample(seed = None)}}.
> {{Sample}} carries no expressions, so {{QueryPlan.deterministic}} only checks
> the child and returns true. {{SampleExec}} draws the seed in its constructor
> through {{Sample.resolveSeed}}, once per physical operator.
> Measured on a 20000-row table, with AQE and broadcast off:
> {code:sql}
> CREATE VIEW vs AS SELECT id FROM t TABLESAMPLE (50 PERCENT);
> SELECT count(DISTINCT id), count(*)
> FROM (SELECT id FROM vs UNION ALL SELECT id FROM vs);
> {code}
> Independent references return about 15000 distinct ids over about 20000 rows
> ({{[14970, 19864]}}, {{[15047, 20052]}}). One shared sample returns about
> 10000 distinct ids ({{[9998, 19996]}}, {{[10060, 20120]}}). The same view
> with {{REPEATABLE (7)}} returns {{[10005, 20010]}} either way, because the
> seed is already fixed on the operator.
> *Scenario 2: aes_encrypt without an IV.*
> {{aes_encrypt(expr, key)}} defaults to GCM and an empty IV. {{AesEncrypt}}
> does not override {{deterministic}}, and its children are ordinary
> expressions, so the flag is true. {{ExpressionImplUtils.aesEncrypt}} calls
> {{SecureRandom}} when the IV is null or empty and prepends that IV to the
> ciphertext.
> {code:sql}
> CREATE VIEW v AS SELECT aes_encrypt(payload, key) AS c FROM t;
> SELECT count(DISTINCT c), count(*)
> FROM (SELECT c FROM v UNION ALL SELECT c FROM v);
> {code}
> Independent references produce two ciphertexts per input row. A shared plan
> produces one.
> h2. Root cause
> {{QueryPlan.deterministic}} is "every expression and every child is
> deterministic". Randomness that is not an expression is invisible to that
> walk.
> * {{Sample}} stores the seed as {{Option[Long]}}, not an expression. {{None}}
> means "draw at execution". {{rand}}, {{randn}}, {{uuid}}, and {{shuffle}} do
> not have this hole: they extend {{Nondeterministic}}, and
> {{ResolveRandomSeed}} writes a concrete seed during analysis.
> * {{AesEncrypt}} is a {{RuntimeReplaceable}} whose replacement is a
> {{StaticInvoke}} of {{aesEncrypt}}. The random IV is inside that Java method.
> {{ReplaceExpressions}} runs before {{InlineCTE}} and substitutes the
> replacement, so an override on {{AesEncrypt}} alone does not reach later
> rules unless the {{StaticInvoke}} is also non-deterministic.
> Call sites that reuse a plan when the flag is true include {{InlineCTE}} (a
> multi-reference CTE is inlined when {{cteDef.deterministic}} is true),
> scalar-subquery merging, SQL-function determinism, and view-to-CTE
> conversion. A special case at one call site leaves the others, and a
> {{Sample}}-only check still accepts {{aes_encrypt}}.
> h2. Proposed fix
> Report the operator non-deterministic at the source, in a dedicated change
> rather than as a special case at one call site.
> * {{Sample.deterministic}} is {{seed.isDefined && child.deterministic}}.
> {{REPEATABLE}} and a programmatic seed stay deterministic, so sharing them
> does not change rows.
> * {{AesEncrypt}} is non-deterministic when the IV is generated, and its
> {{StaticInvoke}} replacement passes {{isDeterministic = false}} in that case.
> An explicit IV stays deterministic.
> Existing {{deterministic}} checks then keep seedless samples and generated-IV
> encryptions from being collapsed. A multi-reference CTE containing either
> evaluates once, matching {{rand()}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
