James Xu created SPARK-60934:
--------------------------------
Summary: [SQL] Treat seedless TABLESAMPLE and aes_encrypt without
an IV as non-deterministic
Key: SPARK-60934
URL: https://issues.apache.org/jira/browse/SPARK-60934
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: James Xu
h2. Problem
Two operators draw fresh randomness at execution but report {{deterministic =
true}}. Rules that trust the flag then compute the operator once and reuse the
result, so two references change from two independent draws to one shared draw.
*Scenario 1: seedless TABLESAMPLE.*
{{TABLESAMPLE}} without {{REPEATABLE}} parses to {{Sample(seed = None)}}.
{{Sample}} carries no expressions, so {{QueryPlan.deterministic}} only checks
the child and returns true. {{SampleExec}} draws the seed in its constructor
through {{Sample.resolveSeed}}, once per physical operator.
Measured on a 20000-row table, with AQE and broadcast off:
{code:sql}
CREATE VIEW vs AS SELECT id FROM t TABLESAMPLE (50 PERCENT);
SELECT count(DISTINCT id), count(*)
FROM (SELECT id FROM vs UNION ALL SELECT id FROM vs);
{code}
Independent references return about 15000 distinct ids over about 20000 rows
({{[14970, 19864]}}, {{[15047, 20052]}}). One shared sample returns about 10000
distinct ids ({{[9998, 19996]}}, {{[10060, 20120]}}). The same view with
{{REPEATABLE (7)}} returns {{[10005, 20010]}} either way, because the seed is
already fixed on the operator.
*Scenario 2: aes_encrypt without an IV.*
{{aes_encrypt(expr, key)}} defaults to GCM and an empty IV. {{AesEncrypt}} does
not override {{deterministic}}, and its children are ordinary expressions, so
the flag is true. {{ExpressionImplUtils.aesEncrypt}} calls {{SecureRandom}}
when the IV is null or empty and prepends that IV to the ciphertext.
{code:sql}
CREATE VIEW v AS SELECT aes_encrypt(payload, key) AS c FROM t;
SELECT count(DISTINCT c), count(*)
FROM (SELECT c FROM v UNION ALL SELECT c FROM v);
{code}
Independent references produce two ciphertexts per input row. A shared plan
produces one.
h2. Root cause
{{QueryPlan.deterministic}} is "every expression and every child is
deterministic". Randomness that is not an expression is invisible to that walk.
* {{Sample}} stores the seed as {{Option[Long]}}, not an expression. {{None}}
means "draw at execution". {{rand}}, {{randn}}, {{uuid}}, and {{shuffle}} do
not have this hole: they extend {{Nondeterministic}}, and {{ResolveRandomSeed}}
writes a concrete seed during analysis.
* {{AesEncrypt}} is a {{RuntimeReplaceable}} whose replacement is a
{{StaticInvoke}} of {{aesEncrypt}}. The random IV is inside that Java method.
{{ReplaceExpressions}} runs before {{InlineCTE}} and substitutes the
replacement, so an override on {{AesEncrypt}} alone does not reach later rules
unless the {{StaticInvoke}} is also non-deterministic.
Call sites that reuse a plan when the flag is true include {{InlineCTE}} (a
multi-reference CTE is inlined when {{cteDef.deterministic}} is true),
scalar-subquery merging, SQL-function determinism, and view-to-CTE conversion.
A special case at one call site leaves the others, and a {{Sample}}-only check
still accepts {{aes_encrypt}}.
h2. Proposed fix
Report the operator non-deterministic at the source, in a dedicated change
rather than as a special case at one call site.
* {{Sample.deterministic}} is {{seed.isDefined && child.deterministic}}.
{{REPEATABLE}} and a programmatic seed stay deterministic, so sharing them does
not change rows.
* {{AesEncrypt}} is non-deterministic when the IV is generated, and its
{{StaticInvoke}} replacement passes {{isDeterministic = false}} in that case.
An explicit IV stays deterministic.
Existing {{deterministic}} checks then keep seedless samples and generated-IV
encryptions from being collapsed. A multi-reference CTE containing either
evaluates once, matching {{rand()}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
