Igor Konev created SPARK-43978: ---------------------------------- Summary: Dropping blocks from memory to disk may result in heartbeat loss Key: SPARK-43978 URL: https://issues.apache.org/jira/browse/SPARK-43978 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.3.2 Reporter: Igor Konev
Dropping blocks from memory to disk is executed inside a synchronized block on MemoryManager (see org.apache.spark.storage.memory.MemoryStore#evictBlocksToFreeSpace). Heartbeats include memory metrics that are retrieved from MemoryManager using synchronized methods. When blocks are being dropped, heartbeats cannot be sent as Heartbeater is blocks. If dropping blocks takes longer than the network timeout, heartbeats are considered lost and the executor gets killed by the driver. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org