Re: Dynamic Resource Allocation with Spark Streaming (Standalone Cluster, Spark 1.5.1)

Till Spark Streaming supports dynamic allocation, you could use StreamingListener to monitor batch execution times and based on it sparkContext.requestExecutors() and sparkContext.killExecutors() to add and remove executors explicitly and . On 26 October 2015 at 21:37, Ted Yu

RE: Dynamic Resource Allocation with Spark Streaming (Standalone Cluster, Spark 1.5.1)

Hi Matthias, Unless there was a change in 1.5, I'm afraid dynamic resource allocation is not yet supported in streaming apps. Thanks, Silvio Sent from my Lumia 930 From: Matthias Niehoff Sent: ‎10/‎26/‎2015 4:00 PM To:

Re: Dynamic Resource Allocation with Spark Streaming (Standalone Cluster, Spark 1.5.1)

This is related: SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs which went into 1.6.0 as well. FYI On Mon, Oct 26, 2015 at 2:26 PM, Silvio Fiorito < silvio.fior...@granturing.com> wrote: > Hi Matthias, > > Unless there was a change in 1.5, I'm afraid dynamic resource