Emil Ejbyfeldt created SPARK-38101: -------------------------------------- Summary: MetadataFetchFailedException due to decommission block migrations Key: SPARK-38101 URL: https://issues.apache.org/jira/browse/SPARK-38101 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.2.1, 3.1.2, 3.1.3, 3.2.2, 3.3 Reporter: Emil Ejbyfeldt
As noted in SPARK-34939 there is race when using broadcast for map output status. Explanation from SPARK-34939 > After map statuses are broadcasted and the executors obtain serialized > broadcasted map statuses. If any fetch failure happens after, Spark scheduler > invalidates cached map statuses and destroy broadcasted value of the map > statuses. Then any executor trying to deserialize serialized broadcasted map > statuses and access broadcasted value, IOException will be thrown. Currently > we don't catch it in MapOutputTrackerWorker and above exception will fail the > application. But if running with `spark.decommission.enabled=true` and `spark.storage.decommission.shuffleBlocks.enabled=true` there is another way to hit this race, when a node is decommissioning and the shuffle blocks are migrated. After a block has been migrated an update will be sent to the driver for each block and the map output caches will be invalidated. Here are a driver when we hit the race condition running with spark 3.2.0: {code:java} 2022-01-28 03:20:12,409 INFO memory.MemoryStore: Block broadcast_27 stored as values in memory (estimated size 5.5 MiB, free 11.0 GiB) 2022-01-28 03:20:12,410 INFO spark.ShuffleStatus: Updating map output for 192108 to BlockManagerId(760, ip-10-231-63-204.ec2.internal, 34707, None) 2022-01-28 03:20:12,410 INFO spark.ShuffleStatus: Updating map output for 179529 to BlockManagerId(743, ip-10-231-34-160.ec2.internal, 44225, None) 2022-01-28 03:20:12,414 INFO spark.ShuffleStatus: Updating map output for 187194 to BlockManagerId(761, ip-10-231-43-219.ec2.internal, 39943, None) 2022-01-28 03:20:12,415 INFO spark.ShuffleStatus: Updating map output for 190303 to BlockManagerId(270, ip-10-231-33-206.ec2.internal, 38965, None) 2022-01-28 03:20:12,416 INFO spark.ShuffleStatus: Updating map output for 192220 to BlockManagerId(270, ip-10-231-33-206.ec2.internal, 38965, None) 2022-01-28 03:20:12,416 INFO spark.ShuffleStatus: Updating map output for 182306 to BlockManagerId(688, ip-10-231-43-41.ec2.internal, 35967, None) 2022-01-28 03:20:12,417 INFO spark.ShuffleStatus: Updating map output for 190387 to BlockManagerId(772, ip-10-231-55-173.ec2.internal, 35523, None) 2022-01-28 03:20:12,417 INFO memory.MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 4.0 MiB, free 10.9 GiB) 2022-01-28 03:20:12,417 INFO storage.BlockManagerInfo: Added broadcast_27_piece0 in memory on ip-10-231-63-1.ec2.internal:34761 (size: 4.0 MiB, free: 11.0 GiB) 2022-01-28 03:20:12,418 INFO memory.MemoryStore: Block broadcast_27_piece1 stored as bytes in memory (estimated size 1520.4 KiB, free 10.9 GiB) 2022-01-28 03:20:12,418 INFO storage.BlockManagerInfo: Added broadcast_27_piece1 in memory on ip-10-231-63-1.ec2.internal:34761 (size: 1520.4 KiB, free: 11.0 GiB) 2022-01-28 03:20:12,418 INFO spark.MapOutputTracker: Broadcast outputstatuses size = 416, actual size = 5747443 2022-01-28 03:20:12,419 INFO spark.ShuffleStatus: Updating map output for 153389 to BlockManagerId(154, ip-10-231-42-104.ec2.internal, 44717, None) 2022-01-28 03:20:12,419 INFO broadcast.TorrentBroadcast: Destroying Broadcast(27) (from updateMapOutput at BlockManagerMasterEndpoint.scala:594) 2022-01-28 03:20:12,427 INFO storage.BlockManagerInfo: Added rdd_65_20310 on disk on ip-10-231-32-25.ec2.internal:40657 (size: 77.6 MiB) 2022-01-28 03:20:12,427 INFO storage.BlockManagerInfo: Removed broadcast_27_piece0 on ip-10-231-63-1.ec2.internal:34761 in memory (size: 4.0 MiB, free: 11.0 GiB) {code} While the Broadcast is being constructed we have updates coming in and the broadcast is destroyed almost immediately. On this particular job we ended up hitting the race condition a lot of times and it caused ~18 task failures and stage retries within 20 seconds causing us to hit our stage retry limit and the job to fail. As far I understand this was the expected behavior for handling this case after SPARK-34939. But it seems like when combined with decommissioning hitting the race is a bit too common. We have observed this behavior running 3.2.0 and 3.2.1, but I think other current versions are all so affected. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org