[
https://issues.apache.org/jira/browse/SPARK-58412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-58412.
------------------------------
Fix Version/s: 4.3.0
5.0.0
Resolution: Fixed
> Incorrect cache statistics after partial materialization or cache rebuild
> -------------------------------------------------------------------------
>
> Key: SPARK-58412
> URL: https://issues.apache.org/jira/browse/SPARK-58412
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0, 5.0.0
>
>
> CachedRDDBuilder can publish incorrect materialization statistics in two
> lifecycle cases.
> First, a task completion listener currently records per-partition row and
> byte counts whenever a task finishes successfully, even if the cache-building
> iterator was not exhausted. This can happen when a MEMORY_ONLY block cannot
> be stored and the partially unrolled iterator is returned to a downstream
> consumer that stops early. Because PartitionKeyedAccumulator is
> last-write-wins per partition, a partial recomputation can replace a complete
> value, such as (10 rows, N bytes), with (0 rows, 0 bytes) while leaving the
> partition key present. InMemoryRelation can therefore still appear fully
> materialized and AQE may treat a non-empty cache as empty.
> Second, clearCache resets the current accumulator in place. Tasks from the
> retired cache generation have already captured that same accumulator. Their
> late completions can write old partition keys and values after the reset,
> contaminating the rebuilt generation and potentially making it appear
> complete before its own partitions finish.
> The fix should:
> * publish partition statistics only after the wrapped iterator is fully
> consumed and the task completes successfully; and
> * install a new PartitionKeyedAccumulator for every cache generation instead
> of resetting the previous accumulator.
> This issue is related to SPARK-57547, which introduced partition-keyed
> materialization bookkeeping and is present in Spark 4.2.0. The fixes apply to
> general InMemoryRelation cache bookkeeping and are independent of runtime
> Bloom filters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
