L. C. Hsieh created SPARK-60112:
-----------------------------------
Summary: Track memory consumers by identity in TaskMemoryManager
Key: SPARK-60112
URL: https://issues.apache.org/jira/browse/SPARK-60112
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
TaskMemoryManager keeps spillable consumers in a value-equality
HashSet<MemoryConsumer>. A consumer whose equals is value-based (e.g. a Scala
case class) is deduplicated against an equal but distinct consumer: the second
instance is never offered for spilling and its memory is missing from the
per-consumer memory breakdown on OOM. SPARK-60071 fixed this for
HybridRowQueue. Tracking consumers in an identity-based set (e.g.
Collections.newSetFromMap(new IdentityHashMap<>())) would protect every
consumer in one place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
