Hey everyone, I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level resource configuration and scheduling
This is for supporting stage level resource configuration and scheduling. The basic idea is to allow the user to specify executor and task resource requirements for each stage to allow the user to control the resources required at a finer grain. One good example here is doing some ETL to preprocess your data in one stage and then feed that data into an ML algorithm (like tensorflow) that would run as a separate stage. The ETL could need totally different resource requirements for the executors/tasks than the ML stage does. The text for the SPIP is in the jira description: https://issues.apache.org/jira/browse/SPARK-27495 I split the API and Design parts into a google doc that is linked to from the jira. This vote is open until next Fri (Sept 13th). [ ] +1: Accept the proposal as an official SPIP [ ] +0 [ ] -1: I don't think this is a good idea because ... I'll start with my +1 Thanks, Tom --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
