[
https://issues.apache.org/jira/browse/SPARK-60945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60945:
-----------------------------------
Labels: pull-request-available (was: )
> from_protobuf/to_protobuf can corrupt a shared options map when the plan is
> canonicalized concurrently
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60945
> URL: https://issues.apache.org/jira/browse/SPARK-60945
> Project: Spark
> Issue Type: Bug
> Components: Protobuf, SQL
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Lihang Shen
> Priority: Major
> Labels: pull-request-available
>
> When a table write triggers a cache refresh check, Spark compares the plan
> for the updated table with cached plans. The comparison uses the
> canonicalized form of both plans. With concurrent writes, different threads
> can end up canonicalizing the same cached plan at the same time.
> If the cached plan contains \{{from_protobuf(..., map(...))}},
> canonicalization can create new \{{FromProtobuf}} expressions when it
> renumbers column IDs. These copies still share the same options expression.
> Canonicalization also evaluates those options. When they use \{{CreateMap}},
> the threads can modify the same internal map builder concurrently, producing
> duplicate keys or mismatched key/value array lengths. This causes
> \{{DUPLICATED_MAP_KEY}} or \{{IllegalArgumentException: requirement failed}}.
> The fix calls \{{options.freshCopyIfContainsStatefulExpression()}} before
> evaluating options in both \{{FromProtobuf}} and \{{ToProtobuf}}, so each
> evaluation uses its own map builder.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
