Till Spark Streaming supports dynamic allocation, you could use StreamingListener to monitor batch execution times and based on it sparkContext.requestExecutors() and sparkContext.killExecutors() to add and remove executors explicitly and .
On 26 October 2015 at 21:37, Ted Yu <yuzhih...@gmail.com> wrote: > This is related: > SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs > > which went into 1.6.0 as well. > > FYI > > On Mon, Oct 26, 2015 at 2:26 PM, Silvio Fiorito < > silvio.fior...@granturing.com> wrote: > >> Hi Matthias, >> >> Unless there was a change in 1.5, I'm afraid dynamic resource allocation >> is not yet supported in streaming apps. >> >> Thanks, >> Silvio >> >> Sent from my Lumia 930 >> ------------------------------ >> From: Matthias Niehoff <matthias.nieh...@codecentric.de> >> Sent: 10/26/2015 4:00 PM >> To: user@spark.apache.org >> Subject: Dynamic Resource Allocation with Spark Streaming (Standalone >> Cluster, Spark 1.5.1) >> >> Hello everybody, >> >> I have a few (~15) Spark Streaming jobs which have load peaks as well as >> long times with a low load. So I thought the new Dynamic Resource >> Allocation for Standalone Clusters might be helpful (SPARK-4751). >> >> I have a test "cluster" with 1 worker consisting of 4 executors with 2 >> cores each, so 8 cores in total. >> >> I started a simple streaming application without limiting the max cores >> for this app. As expected the app occupied every core of the cluster. Then >> I started a second app, also without limiting the maximum cores. As the >> first app did not get any input through the stream, my naive expectation >> was that the second app would get at least 2 cores (1 receiver, 1 >> processing), but that's not what happened. The cores are still assigned to >> the first app. >> When I look at the application UI of the first app every executor is >> still running. That explains why no executor is used for the second app. >> >> I end up with two questions: >> - When does an executor getting idle in a Spark Streaming application? >> (and so could be reassigned to another app) >> - Is there another way to compete with uncertain load when using Spark >> Streaming Applications? I already combined multiple jobs to a Spark >> Application using different threads, but this approach comes to a limit for >> me, because Spark Applications get to big to manage. >> >> Thank You! >> >> >> >
