Hey everyone, I have been working on coming up with a proposal for supporting stage level resource configuration and scheduling. The basic idea is to allow the user to specify executor and task resource requirements for each stage to allow the user to control the resources required at a finer grain. One good example here is doing some ETL to preprocess your data in one stage and then feed that data into an ML algorithm (like tensorflow) that would run as a separate stage. The ETL could need totally different resource requirements for the executors/tasks than the ML stage does. If you are interested please take a look at the SPIP and give me feedback. The text for the SPIP is in the jira description: https://issues.apache.org/jira/browse/SPARK-27495
I split the API and Design parts into a google doc that is linked to from the jira. Thanks,Tom