Xianjin YE created SPARK-24006: ---------------------------------- Summary: ExecutorAllocationManager.onExecutorAdded is an O(n) operation Key: SPARK-24006 URL: https://issues.apache.org/jira/browse/SPARK-24006 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.3.0 Reporter: Xianjin YE
The ExecutorAllocationManager.onExecutorAdded is an O(n) operations, I believe it will be a problem when scaling out with large number of Executors as it effectively makes adding N executors at time complexity O(N^2). I propose to invoke onExecutorIdle guarded by {code:java} if (executorIds.size - executorsPendingToRemove.size >= minNumExecutors +1) { // Since we only need to re-remark idle executors when low bound executorIds.filter(listener.isExecutorIdle).foreach(onExecutorIdle) }{code} cc [~zsxwing] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org