[jira] [Updated] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12133: -- Target Version/s: 2.0.0 > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. The PR will revert the changes added in SPARK-10955, > which warns against using dynamic allocation with streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12133: -- Attachment: dynamic-allocation-streaming-design.pdf > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or >Assignee: Tathagata Das > Attachments: dynamic-allocation-streaming-design.pdf > > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. A design doc will be posted shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12133: -- Description: Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons: (1) Your executors may never be idle since they run something every N seconds (2) You should have at least one receiver running always (3) The existing heuristics don't take into account length of batch queue ... The goal of this JIRA is to provide better support for using dynamic allocation in streaming. A design doc will be posted shortly. was: Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons: (1) Your executors may never be idle since they run something every N seconds (2) You should have at least one receiver running always (3) The existing heuristics don't take into account length of batch queue ... The goal of this JIRA is to provide better support for using dynamic allocation in streaming. > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. A design doc will be posted shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12133: -- Assignee: Tathagata Das > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or >Assignee: Tathagata Das > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. A design doc will be posted shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12133: -- Description: Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons: (1) Your executors may never be idle since they run something every N seconds (2) You should have at least one receiver running always (3) The existing heuristics don't take into account length of batch queue ... The goal of this JIRA is to provide better support for using dynamic allocation in streaming. was: Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons: (1) Your executors may never be idle since they run something every N seconds (2) You should have at least one receiver running always (3) The existing heuristics don't take into account length of batch queue ... The goal of this JIRA is to provide better support for using dynamic allocation in streaming. The PR will revert the changes added in SPARK-10955, which warns against using dynamic allocation with streaming. > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
