[
https://issues.apache.org/jira/browse/SPARK-58209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58209:
-----------------------------------
Labels: pull-request-available (was: )
> Mark XPathExtract, CallMethodViaReflection, and UserDefinedGenerator stateful
> to prevent executor-side data races
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58209
> URL: https://issues.apache.org/jira/browse/SPARK-58209
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.5.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0
> Reporter: Vinod KC
> Priority: Major
> Labels: pull-request-available
>
> Three Catalyst expressions `XPathExtract` `CallMethodViaReflection` ,
> `UserDefinedGenerator` maintain mutable state internally but do not override
> stateful (which defaults to false).
> As a result, freshCopyIfContainsStatefulExpression() never makes independent
> copies of them. When two executor tasks share the same expression instance —
> which can happen when sibling QueryExecutions are derived from the same base
> DataFrame — they race on the shared mutable fields and silently produce
> incorrect results.
>
> Need to override def stateful: Boolean = true to XPathExtract,
> CallMethodViaReflection, and UserDefinedGenerator. This will cause
> freshCopyIfContainsStatefulExpression() to return a fresh instance per
> concurrent task, giving each its own independent evaluator, buffer, and lazy
> vars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
