Github user mariahualiu commented on the issue: https://github.com/apache/spark/pull/17854 Let me describe what I've seen when using 2500 executors. 1. In the first a few (2~3) requests, AM received all (in this case 2500) containers from Yarn. 2. In a few seconds, 2500 launch container commands were sent out. 3. It took 3~4 minutes to start an executor on an NM (most of the time was spent on container localization: downloading spark jar, application jar and etc. from the hdfs staging folder). 4. A large number of executors tried to retrieve spark properties from driver but failed to connect. A massive removing failed executors happened. It seems to me RemoveExecutor is handled by the same single thread that responds to RetrieveSparkProps and RegisterExecutor. As a result, this thread was even busier, and more executors cannot connect/register/etc. 5. YarnAllocator requested more containers to make up for the failed ones. More executors tried to retrieve spark props and register. However the thread was still overwhelmed by the previous round of executors and cannot respond. In some cases, we got 5000 executor failures and the application retried and eventually failed.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org