So receivers will be fixed for every run of streaming interval job. Say I
have set stream Duration to be 10 minutes, then after each 10 minute job
will be created and same executor nodes say in your
case(spark-akhil-slave2.c.neat-axis-616.internal
and spark-akhil-slave1.c.neat-axis-616.internal) will be used ?

Is it with all streaming applications that executor nodes will be fixed and
won't change depending on load of current batch or is it for kafka case
only??









On Wed, Apr 15, 2015 at 4:12 PM, Akhil Das <[email protected]>
wrote:

> Once you start your streaming application to read from Kafka, it will
> launch receivers on the executor nodes. And you can see them on the
> streaming tab of your driver ui (runs on 4040).
>
> [image: Inline image 1]
>
> These receivers will be fixed till the end of your pipeline (unless its
> crashed etc.) You can say, eah receiver will run on a single core.
>
> Thanks
> Best Regards
>
> On Wed, Apr 15, 2015 at 3:46 PM, Shushant Arora <[email protected]
> > wrote:
>
>> Hi
>>
>> I want to understand the flow of spark streaming with kafka.
>>
>> In spark Streaming is the executor nodes at each run of streaming
>> interval same or At each stream interval cluster manager assigns new
>> executor nodes for processing this batch input. If yes then at each batch
>> interval new executors register themselves as kafka consumers?
>>
>> Even without kafka is executor nodes on each batch interval same or
>> driver nodes gets new executor nodes from cluster manager ?
>>
>> Thanks
>> Shushant
>>
>
>

Reply via email to