Re: Dynamic Allocation & Spark Streaming

It's has been a while since last update on the thread. Now Spark 2.0 is available, do you guys know if there's any progress on Dynamic Allocation & Spark Streaming? On Mon, Oct 19, 2015 at 1:13 PM, robert towne <binarymecha...@gmail.com> wrote: > I have watched a few videos fr

Re: Dynamic Allocation & Spark Streaming

://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.scheduler.BatchInfo From: Kyle Lin Date: Friday, November 6, 2015 at 11:48 AM To: Tathagata Das Cc: robert towne, user Subject: Re: Dynamic Allocation & Spark Streaming Hey there I run Spark streaming 1.5.1 on

Re: Dynamic Allocation & Spark Streaming

Hey there I run Spark streaming 1.5.1 on YARN with Dynamic allocation, and use direct stream API to read data from Kafka. Spark job can dynamically request a executor when reaching spark.dynamicAllocation.schedulerBacklogTimeout. However, it won't dynamically remove executor when there is no

Dynamic Allocation & Spark Streaming

I have watched a few videos from Databricks/Andrew Or around the Spark 1.2 release and it seemed that dynamic allocation was not yet available for Spark Streaming. I now see SPARK-10955 which is tied to 1.5.2 and allows disabling of Spark

Dynamic Allocation & Spark Streaming

I have watched a few videos from Databricks/Andrew Or around the Spark 1.2 release and it seemed that dynamic allocation was not yet available for Spark Streaming. I now see SPARK-10955 which is tied to 1.5.2 and allows disabling of Spark

Re: Dynamic Allocation & Spark Streaming

Unfortunately the title on the JIRA is extremely confusing. I have fixed it. The reason why dynamic allocation does not work well with streaming is that the heuristic that is used to automatically scale up or down the number of executors works for the pattern of task schedules in batch jobs, not