[ 
https://issues.apache.org/jira/browse/SPARK-61419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lihang Shen updated SPARK-61419:
--------------------------------
    Description: 
Same issue as SPARK-60945. \{{FromAvro.replacement}} and 
\{{SchemaOfAvro.replacement}} evaluate the
stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent 
canonicalizations of one plan share
that map and race on its map builder, failing with 
\{{IllegalArgumentException}} or
{\{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix 
is the same:
evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.

  was:
{{Same issue as SPARK-60945. \{{FromAvro.replacement}} and 
\{{SchemaOfAvro.replacement}} evaluate the
stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent 
canonicalizations of one plan share
that map and race on its map builder, failing with 
\{{IllegalArgumentException}} or
\{{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix 
is the same:
evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.}}


> from_avro/schema_of_avro can corrupt a shared options map when the plan is 
> canonicalized concurrently
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61419
>                 URL: https://issues.apache.org/jira/browse/SPARK-61419
>             Project: Spark
>          Issue Type: Bug
>          Components: Bug
>    Affects Versions: 4.0.0
>            Reporter: Lihang Shen
>            Priority: Major
>
> Same issue as SPARK-60945. \{{FromAvro.replacement}} and 
> \{{SchemaOfAvro.replacement}} evaluate the
> stateful \{{options}} map (\{{CreateMap}}) directly. Concurrent 
> canonicalizations of one plan share
> that map and race on its map builder, failing with 
> \{{IllegalArgumentException}} or
> {\{DUPLICATED_MAP_KEY}}, or silently dropping options under LAST_WIN. The fix 
> is the same:
> evaluate \{{options.freshCopyIfContainsStatefulExpression()}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to