tgravescs commented on pull request #29332: URL: https://github.com/apache/spark/pull/29332#issuecomment-670010625
Oh right, if you didn't specify the executor cores in standalone mode - you get them all by default. Which honestly causes lots of issues - I filed a jira for this for other things. that is left up to the user to properly configure their cluster and job. There are other things in spark 3.0 that expect cores to be the limiting resource. Dynamic allocation is a big one, which I know you don't care about, but I'm pretty sure there are other things throughout the code that does as well. We should remove the isDynamicAllocationCheck here so that the warning gets printed for standalone as well: https://github.com/apache/spark/blob/branch-3.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L2836 If you want to go through and make sure everything is updated to schedule based on resources, I'm fine with it but I'm pretty sure needs to be more then this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org