[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-8873: ----------------------------------- Assignee: Timothy Chen (was: Apache Spark) > Support cleaning up shuffle files when using shuffle service in Mesos > --------------------------------------------------------------------- > > Key: SPARK-8873 > URL: https://issues.apache.org/jira/browse/SPARK-8873 > Project: Spark > Issue Type: Improvement > Components: Mesos > Affects Versions: 1.2.0 > Reporter: Timothy Chen > Assignee: Timothy Chen > Priority: Critical > Labels: mesos > > With dynamic allocation enabled with Mesos, drivers can launch with shuffle > data cached in the external shuffle service. > However, there is no reliable way to let the shuffle service clean up the > shuffle data when the driver exits, since it may crash before it notifies the > shuffle service and shuffle data will be cached forever. > We need to implement a reliable way to detect driver termination and clean up > shuffle data accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org