[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442735#comment-16442735 ]
Hyukjin Kwon commented on SPARK-24006: -------------------------------------- Would be good to make a small test with the logic above and see if how much it improves. If that's ignorable in practice, I would leave this resolved. > ExecutorAllocationManager.onExecutorAdded is an O(n) operation > -------------------------------------------------------------- > > Key: SPARK-24006 > URL: https://issues.apache.org/jira/browse/SPARK-24006 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Xianjin YE > Priority: Major > > The ExecutorAllocationManager.onExecutorAdded is an O(n) operations, I > believe it will be a problem when scaling out with large number of Executors > as it effectively makes adding N executors at time complexity O(N^2). > > I propose to invoke onExecutorIdle guarded by > {code:java} > if (executorIds.size - executorsPendingToRemove.size >= minNumExecutors +1) { > // Since we only need to re-remark idle executors when low bound > executorIds.filter(listener.isExecutorIdle).foreach(onExecutorIdle) > } else { > onExecutorIdle(executorId) > }{code} > cc [~zsxwing] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org