[
https://issues.apache.org/jira/browse/SPARK-61419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lihang Shen updated SPARK-61419:
--------------------------------
Description:
Same issue as SPARK-60945. \{{FromAvro.replacement}} and
\{{SchemaOfAvro.replacement}} evaluate the
stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent
canonicalizations of one plan share
that map and race on its map builder, failing with
\{{IllegalArgumentException}} or
{\{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix
is the same:
evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.
was:
{{Same issue as SPARK-60945. \{{FromAvro.replacement}} and
\{{SchemaOfAvro.replacement}} evaluate the
stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent
canonicalizations of one plan share
that map and race on its map builder, failing with
\{{IllegalArgumentException}} or
\{{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix
is the same:
evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.}}
> from_avro/schema_of_avro can corrupt a shared options map when the plan is
> canonicalized concurrently
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61419
> URL: https://issues.apache.org/jira/browse/SPARK-61419
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.0.0
> Reporter: Lihang Shen
> Priority: Major
>
> Same issue as SPARK-60945. \{{FromAvro.replacement}} and
> \{{SchemaOfAvro.replacement}} evaluate the
> stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent
> canonicalizations of one plan share
> that map and race on its map builder, failing with
> \{{IllegalArgumentException}} or
> {\{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix
> is the same:
> evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]