[jira] [Commented] (SPARK-14723) A new way to support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247435#comment-15247435 ] WilliamZhu commented on SPARK-14723: I think it would be better to give extra executors the first time ,then reduce the number gradually. But the reduce action should not be too slow cause sometimes we have massive executors. the current behavior – adding executors when there's work to do - is not ideal for streaming application especially those with a short duration cause adding executors is slow job. Also,you should consider the situation that the cluster have resource starving temporarily . We will request the number of spark.streaming.dynamicAllocation.maxExecutors executors from Yarn immediately in greedy way once there is any delay happens ,since we hope we can eliminate the delay as soon as possible .Then we can reduce the redundant executors. > A new way to support dynamic allocation in Spark Streaming > -- > > Key: SPARK-14723 > URL: https://issues.apache.org/jira/browse/SPARK-14723 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Streaming >Reporter: WilliamZhu > Attachments: spark-streaming-dynamic-allocation-desigh.pdf > > > Provide a more powerful Algorithm to support dynamic allocation in spark > streaming. > more details: http://www.jianshu.com/p/ae7fdd4746f6 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14723) A new way to support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WilliamZhu updated SPARK-14723: --- Attachment: spark-streaming-dynamic-allocation-desigh.pdf > A new way to support dynamic allocation in Spark Streaming > -- > > Key: SPARK-14723 > URL: https://issues.apache.org/jira/browse/SPARK-14723 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Streaming >Reporter: WilliamZhu > Labels: features > Fix For: 2.1.0 > > Attachments: spark-streaming-dynamic-allocation-desigh.pdf > > > Provide a more powerful Algorithm to support dynamic allocation in spark > streaming. > more details: http://www.jianshu.com/p/ae7fdd4746f6 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-14723) A new way to support dynamic allocation in Spark Streaming
WilliamZhu created SPARK-14723: -- Summary: A new way to support dynamic allocation in Spark Streaming Key: SPARK-14723 URL: https://issues.apache.org/jira/browse/SPARK-14723 Project: Spark Issue Type: Improvement Components: Spark Core, Streaming Reporter: WilliamZhu Fix For: 2.1.0 Provide a more powerful Algorithm to support dynamic allocation in spark streaming. more details: http://www.jianshu.com/p/ae7fdd4746f6 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12133) Support dynamic allocation in Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240453#comment-15240453 ] WilliamZhu commented on SPARK-12133: Here have a new Design: http://www.jianshu.com/p/ae7fdd4746f6 > Support dynamic allocation in Spark Streaming > - > > Key: SPARK-12133 > URL: https://issues.apache.org/jira/browse/SPARK-12133 > Project: Spark > Issue Type: Bug > Components: Spark Core, Streaming >Reporter: Andrew Or >Assignee: Tathagata Das > Attachments: dynamic-allocation-streaming-design.pdf > > > Dynamic allocation is a feature that allows your cluster resources to scale > up and down based on the workload. Currently it doesn't work well with Spark > streaming because of several reasons: > (1) Your executors may never be idle since they run something every N seconds > (2) You should have at least one receiver running always > (3) The existing heuristics don't take into account length of batch queue > ... > The goal of this JIRA is to provide better support for using dynamic > allocation in streaming. A design doc will be posted shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
