Vinod KC created SPARK-58204:
--------------------------------
Summary: Fix unprotected mutable state in RegExpReplace,
RegExpExtractBase, StringTranslate, FormatNumber, and NamedLambdaVariable by
marking them stateful
Key: SPARK-58204
URL: https://issues.apache.org/jira/browse/SPARK-58204
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0, 4.0.0, 4.1.0, 3.5.0, 4.3.0
Reporter: Vinod KC
Several Catalyst expressions maintain mutable state in @transient instance
fields but do not override `stateful`, so
`freshCopyIfContainsStatefulExpression()` never copies them.
When the Several Catalyst expressions maintain mutable state in @transient
instance fields but do not override `stateful`, so
`freshCopyIfContainsStatefulExpression()` never copies them.
When the same expression node is shared across multiple projections (e.g. via
ConvertToLocalRelation or pipelined execution), threads race on the shared
cache, corrupting results silently.
Affected expressions and their mutable state:
- RegExpReplace: lastRegex, pattern, lastReplacement, lastReplacementInUTF8
- RegExpExtractBase (RegExpExtract, RegExpExtractAll): lastRegex, pattern
- StringTranslate: lastMatching, lastReplace, dict
- FormatNumber: lastDIntValue, lastDStringValue, numberFormat (DecimalFormat)
- NamedLambdaVariable: value (AtomicReference[Any])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
