Chao Sun created SPARK-58412:
--------------------------------
Summary: Incorrect cache statistics after partial materialization
or cache rebuild
Key: SPARK-58412
URL: https://issues.apache.org/jira/browse/SPARK-58412
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
CachedRDDBuilder can publish incorrect materialization statistics in two
lifecycle cases.
First, a task completion listener currently records per-partition row and byte
counts whenever a task finishes successfully, even if the cache-building
iterator was not exhausted. This can happen when a MEMORY_ONLY block cannot be
stored and the partially unrolled iterator is returned to a downstream consumer
that stops early. Because PartitionKeyedAccumulator is last-write-wins per
partition, a partial recomputation can replace a complete value, such as (10
rows, N bytes), with (0 rows, 0 bytes) while leaving the partition key present.
InMemoryRelation can therefore still appear fully materialized and AQE may
treat a non-empty cache as empty.
Second, clearCache resets the current accumulator in place. Tasks from the
retired cache generation have already captured that same accumulator. Their
late completions can write old partition keys and values after the reset,
contaminating the rebuilt generation and potentially making it appear complete
before its own partitions finish.
The fix should:
* publish partition statistics only after the wrapped iterator is fully
consumed and the task completes successfully; and
* install a new PartitionKeyedAccumulator for every cache generation instead of
resetting the previous accumulator.
This issue is related to SPARK-57547, which introduced partition-keyed
materialization bookkeeping and is present in Spark 4.2.0. The fixes apply to
general InMemoryRelation cache bookkeeping and are independent of runtime Bloom
filters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
