Aniket Bhatnagar created SPARK-18421: ----------------------------------------
Summary: Dynamic disk allocation Key: SPARK-18421 URL: https://issues.apache.org/jira/browse/SPARK-18421 Project: Spark Issue Type: New Feature Components: Spark Core Affects Versions: 2.0.1 Reporter: Aniket Bhatnagar Priority: Minor Dynamic allocation feature allows you to add executors and scale computation power. This is great, however, I feel like we also need a way to dynamically scale storage. Currently, if the disk is not able to hold the spilled/shuffle data, the job is aborted (in yarn, the node manager kills the container) causing frustration and loss of time. In deployments like AWS EMR, it is possible to run an agent that add disks on the fly if it sees that the disks are running out of space and it would be great if Spark could immediately start using the added disks just as it does when new executors are added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org