Re: what's local[n]

[Andrew Ash]

Hi Weide,

The answer to your first question about local[2] can be found in the
"Running the Examples and Shell" section of
https://spark.apache.org/docs/latest/

Note that all of the sample programs take a  parameter specifying
> the cluster URL to connect to. This can be a URL for a distributed 
> cluster,
> or local to run locally with one thread, or local[N] to run locally with
> N threads. You should start by using local for testing.


For the second, I'm not sure how the thread count affects the number of
kafka consumers.  My first guess would be that a thread is 1:1 with a kafka
consumer, but I'm not super familiar with Spark streaming.

Andrew



On Sat, May 3, 2014 at 7:00 AM, Weide Zhang  wrote:

> in spark kafka example,
>
> it says
>
>`./bin/run-example org.apache.spark.streaming.examples.KafkaWordCount
> local[2] zoo01,zoo02,zoo03 my-consumer-group topic1,topic2 1`
>
> can any one tell me what does local[2] represent ? i thought master url
> should be sth like spark://hostname:portname .
>
> also, the thread count is specified as 1 in kafka example, what will
> happen thread count goes to more than 1 ? does that mean multiple kafka
> consumer will be created on different spark workers ? I'm not sure how does
> the code mapped to realtime worker thread allocation ? Is there any
> documentation on that ?
>
> Weide
>

what's local[n]

[Weide Zhang]

in spark kafka example,

it says

   `./bin/run-example org.apache.spark.streaming.examples.KafkaWordCount
local[2] zoo01,zoo02,zoo03 my-consumer-group topic1,topic2 1`

can any one tell me what does local[2] represent ? i thought master url
should be sth like spark://hostname:portname .

also, the thread count is specified as 1 in kafka example, what will happen
thread count goes to more than 1 ? does that mean multiple kafka consumer
will be created on different spark workers ? I'm not sure how does the code
mapped to realtime worker thread allocation ? Is there any documentation on
that ?

Weide