[ https://issues.apache.org/jira/browse/SPARK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268826#comment-14268826 ]
Jongyoul Lee commented on SPARK-4922: ------------------------------------- [~andrewor14] Hi, I have a basic question about your idea. I'm using fine-grained mesos for running my jobs. that mode already allocate resources dynamically when task scheduler wants. What you think the difference is between your idea and fine-grained mode? Unlike coarse-grained mode, fine-grained mode adjusts # of cores for a executor and enables to make two more executor on each slave. I think if we set # of cores for each mesos executor in a configuration on fine-grained mode - now, only one core fixed for each executor -, we can satisfy dynamic allocation idea. and I read SPARK-4751, and I'll handle this issue via using fine-grain mode. And how do you think you adjust resources? new API for increasing or decreasing cores or just use {{spark.cores.max}}? > Support dynamic allocation for coarse-grained Mesos > --------------------------------------------------- > > Key: SPARK-4922 > URL: https://issues.apache.org/jira/browse/SPARK-4922 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 1.2.0 > Reporter: Andrew Or > Priority: Critical > > This brings SPARK-3174, which provided dynamic allocation of cluster > resources to Spark on YARN applications, to Mesos coarse-grained mode. > Note that the translation is not as trivial as adding a code path that > exposes the request and kill mechanisms as we did for YARN is SPARK-3822. > This is because Mesos coarse-grained mode schedules on the notion of setting > the number of cores allowed for an application (as in standalone mode) > instead of number of executors (as in YARN mode). For more detail, please see > SPARK-4751. > If you intend to work on this, please provide a detailed design doc! -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org