Dynamic allocation Spark Stremaing

Hi, I have enabled dynamic allocation for spark streaming application but the number of containers always shows as 2. Is there a way to get more when job is running? Thanks, Asmath

Re: Dynamic Allocation & Spark Streaming

It's has been a while since last update on the thread. Now Spark 2.0 is available, do you guys know if there's any progress on Dynamic Allocation & Spark Streaming? On Mon, Oct 19, 2015 at 1:13 PM, robert towne <binarymecha...@gmail.com> wrote: > I have watched a few videos fr

Re: Dynamic allocation Spark

>> spark.shuffle.service.enabled true >> >> When I launch my application it just stays in the queue accepted but it >> never actually runs. >> 16/02/26 11:11:46 INFO yarn.Client: Application report for >> application_1456482268159_0001 (state: ACCEPTED) >&

Re: Dynamic allocation Spark

it just stays in the queue accepted but it > never actually runs. > 16/02/26 11:11:46 INFO yarn.Client: Application report for > application_1456482268159_0001 (state: ACCEPTED) > > Am I missing something? > > Thanks in advance as always > > > > -- > View this me

Dynamic allocation Spark

11:46 INFO yarn.Client: Application report for application_1456482268159_0001 (state: ACCEPTED) Am I missing something? Thanks in advance as always -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dynamic-allocation-Spark-tp26344.html Sent from the Apache

Re: Dynamic Allocation & Spark Streaming

://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.scheduler.BatchInfo From: Kyle Lin Date: Friday, November 6, 2015 at 11:48 AM To: Tathagata Das Cc: robert towne, user Subject: Re: Dynamic Allocation & Spark Streaming Hey there I run Spark streaming 1.5.1 on

Re: Dynamic Allocation & Spark Streaming

Hey there I run Spark streaming 1.5.1 on YARN with Dynamic allocation, and use direct stream API to read data from Kafka. Spark job can dynamically request a executor when reaching spark.dynamicAllocation.schedulerBacklogTimeout. However, it won't dynamically remove executor when there is no

Dynamic Allocation & Spark Streaming

I have watched a few videos from Databricks/Andrew Or around the Spark 1.2 release and it seemed that dynamic allocation was not yet available for Spark Streaming. I now see SPARK-10955 which is tied to 1.5.2 and allows disabling of Spark

Dynamic Allocation & Spark Streaming

I have watched a few videos from Databricks/Andrew Or around the Spark 1.2 release and it seemed that dynamic allocation was not yet available for Spark Streaming. I now see SPARK-10955 which is tied to 1.5.2 and allows disabling of Spark

Re: Dynamic Allocation & Spark Streaming

Unfortunately the title on the JIRA is extremely confusing. I have fixed it. The reason why dynamic allocation does not work well with streaming is that the heuristic that is used to automatically scale up or down the number of executors works for the pattern of task schedules in batch jobs, not