jerrypeng commented on code in PR #42352: URL: https://github.com/apache/spark/pull/42352#discussion_r1566272683
########## core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala: ########## @@ -669,6 +753,27 @@ private[spark] class ExecutorAllocationManager( private val stageAttemptToExecutorPlacementHints = new mutable.HashMap[StageAttempt, (Int, Map[String, Int], Int)] + // to track total no. of tasks in each stage of a micro-batch (streaming use case) + // this will help in requesting resources by counting pending tasks in job, + // rather than counting pending tasks in a stage. + private val stageIdToNumTasks = new mutable.HashMap[Int, Int] Review Comment: Are there any race conditions for updating this map? I am not an expert on the DAGScheduler or ExecutorAllocationManager, but it seems to be more then one thread will be reading/updating this map. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org