[
https://issues.apache.org/jira/browse/SPARK-60945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lihang Shen updated SPARK-60945:
--------------------------------
Description:
{\{FromProtobuf.replacement}} and \{{ToProtobuf.replacement}} evaluate
\{{options}} directly. When options come from an expression like \{{map('mode',
'PERMISSIVE')}}, this reuses the mutable map builder inside \{{CreateMap}}.
When plan canonicalization changes attribute IDs, it creates a new protobuf
expression but keeps the same options child. Each copy has its own
\{{replacement}} lazy-val lock, so two threads can still evaluate the shared
\{{CreateMap}} at the same time. Both modify the same
\{{ArrayBasedMapBuilder}}, which can cause \{{DUPLICATED_MAP_KEY}} or
\{{IllegalArgumentException: requirement failed}} in \{{ArrayBasedMapData}}.
> from_protobuf/to_protobuf can corrupt a shared options map when the plan is
> canonicalized concurrently
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60945
> URL: https://issues.apache.org/jira/browse/SPARK-60945
> Project: Spark
> Issue Type: Bug
> Components: Protobuf, SQL
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Lihang Shen
> Priority: Major
>
> {\{FromProtobuf.replacement}} and \{{ToProtobuf.replacement}} evaluate
> \{{options}} directly. When options come from an expression like
> \{{map('mode', 'PERMISSIVE')}}, this reuses the mutable map builder inside
> \{{CreateMap}}.
> When plan canonicalization changes attribute IDs, it creates a new protobuf
> expression but keeps the same options child. Each copy has its own
> \{{replacement}} lazy-val lock, so two threads can still evaluate the shared
> \{{CreateMap}} at the same time. Both modify the same
> \{{ArrayBasedMapBuilder}}, which can cause \{{DUPLICATED_MAP_KEY}} or
> \{{IllegalArgumentException: requirement failed}} in \{{ArrayBasedMapData}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
