[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442781#comment-16442781 ]
Xianjin YE commented on SPARK-24006: ------------------------------------ Could you leave this issue open for a while? In my company, we are trying to scaling Spark Application with large data and a lot of executors in the near future. I will report back when I had first experience how this impact dynamic allocation. > ExecutorAllocationManager.onExecutorAdded is an O(n) operation > -------------------------------------------------------------- > > Key: SPARK-24006 > URL: https://issues.apache.org/jira/browse/SPARK-24006 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Xianjin YE > Priority: Major > > The ExecutorAllocationManager.onExecutorAdded is an O(n) operations, I > believe it will be a problem when scaling out with large number of Executors > as it effectively makes adding N executors at time complexity O(N^2). > > I propose to invoke onExecutorIdle guarded by > {code:java} > if (executorIds.size - executorsPendingToRemove.size >= minNumExecutors +1) { > // Since we only need to re-remark idle executors when low bound > executorIds.filter(listener.isExecutorIdle).foreach(onExecutorIdle) > } else { > onExecutorIdle(executorId) > }{code} > cc [~zsxwing] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org