[ https://issues.apache.org/jira/browse/SPARK-43510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manu Zhang updated SPARK-43510: ------------------------------- Description: I see application hangs when containers are preempted immediately after allocation as follows. {code:java} 23/05/14 09:11:33 INFO YarnAllocator: Launching container container_e3812_1684033797982_57865_01_000382 on host hdc42-mcc10-01-0910-4207-015-tess0028.stratus.rno.ebay.com for executor with ID 277 for ResourceProfile Id 0 23/05/14 09:11:33 WARN YarnAllocator: Cannot find executorId for container: container_e3812_1684033797982_57865_01_000382 23/05/14 09:11:33 INFO YarnAllocator: Completed container container_e3812_1684033797982_57865_01_000382 (state: COMPLETE, exit status: -102) 23/05/14 09:11:33 INFO YarnAllocator: Container container_e3812_1684033797982_57865_01_000382 was preempted.{code} Note the warning log where YarnAllocator cannot find executorId for the container when processing completed containers. The only plausible cause is YarnAllocator added the running executor after processing completed containers. The former happens in a separate thread after executor launch. YarnAllocator believes there are still running executors, although they are already lost due to preemption. Hence, the application hangs without any running executors. was: I see application hangs when containers are preempted immediately after allocation as follows. {code:java} 23/05/14 09:11:33 INFO YarnAllocator: Launching container container_e3812_1684033797982_57865_01_000382 on host hdc42-mcc10-01-0910-4207-015-tess0028.stratus.rno.ebay.com for executor with ID 277 for ResourceProfile Id 0 23/05/14 09:11:33 WARN YarnAllocator: Cannot find executorId for container: container_e3812_1684033797982_57865_01_000382 23/05/14 09:11:33 INFO YarnAllocator: Completed container container_e3812_1684033797982_57865_01_000382 (state: COMPLETE, exit status: -102) 23/05/14 09:11:33 INFO YarnAllocator: Container container_e3812_1684033797982_57865_01_000382 was preempted.{code} Note the warning log where YarnAllocator cannot find executorId for the container when processing completed containers. The only plausible cause is YarnAllocator processing completed container before updating internal state and adding the executorId. The latter happens in a separate thread after executor launch. YarnAllocator thought > Spark application hangs when YarnAllocator adding running executors after > processing completed containers > --------------------------------------------------------------------------------------------------------- > > Key: SPARK-43510 > URL: https://issues.apache.org/jira/browse/SPARK-43510 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 3.4.0 > Reporter: Manu Zhang > Priority: Major > > I see application hangs when containers are preempted immediately after > allocation as follows. > {code:java} > 23/05/14 09:11:33 INFO YarnAllocator: Launching container > container_e3812_1684033797982_57865_01_000382 on host > hdc42-mcc10-01-0910-4207-015-tess0028.stratus.rno.ebay.com for executor with > ID 277 for ResourceProfile Id 0 > 23/05/14 09:11:33 WARN YarnAllocator: Cannot find executorId for container: > container_e3812_1684033797982_57865_01_000382 > 23/05/14 09:11:33 INFO YarnAllocator: Completed container > container_e3812_1684033797982_57865_01_000382 (state: COMPLETE, exit status: > -102) > 23/05/14 09:11:33 INFO YarnAllocator: Container > container_e3812_1684033797982_57865_01_000382 was preempted.{code} > Note the warning log where YarnAllocator cannot find executorId for the > container when processing completed containers. The only plausible cause is > YarnAllocator added the running executor after processing completed > containers. The former happens in a separate thread after executor launch. > YarnAllocator believes there are still running executors, although they are > already lost due to preemption. Hence, the application hangs without any > running executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org