Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 Sorry if I was unclear earlier on the issue w/ the active Job ID. So, I agree, that if a user actually gets into this situation, where they've got two different jobs for the same stage, with different max concurrent tasks, its mostly a toss-up which one they'll get, as the users jobs are probably racing to get to that stage. Still, I think its important that it pulls the max concurrent tasks from the active job, just so that users can understand what is going on and for consistency and debugability. The TaskSetManager gets the property from the active job, which actually submitted the stage, so the ExecutorAllocationManager should do the same. I think the best way to ensure that is to add activeJobId to SparkListenerStageSubmitted. Then you'd go back to just keeping a jobIdToMaxConcurrentTasks map when handling onJobStart, and in onStageSubmitted, you'd then figure out the max number of tasks for that stage, given the job which actually submitted it. @tgravescs what do you think?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org