Hi, Akhil,
Thank you for the explanation!
bit1...@163.com
From: Akhil Das
Date: 2015-06-23 16:29
To: bit1...@163.com
CC: user
Subject: Re: What does [Stage 0:> (0 + 2) / 2] mean on the console
Well, you could that (Stage information) is an ASCII representation of the
WebUI (running on port 4040). Since you set local[4] you will have 4 threads
for your computation, and since you are having 2 receivers, you are left with 2
threads to process ((0 + 2) <-- This 2 is your 2 threads.) And the other /2
means you are having 2 tasks in that stage (with id 0).
Thanks
Best Regards
On Tue, Jun 23, 2015 at 1:21 PM, bit1...@163.com <bit1...@163.com> wrote:
Hi,
I have a spark streaming application that runs locally with two receivers, some
code snippet is as follows:
conf.setMaster("local[4]")
//RPC Log Streaming
val rpcStream = KafkaUtils.createStream[String, String, StringDecoder,
StringDecoder](ssc, consumerParams, topicRPC, StorageLevel.MEMORY_ONLY)
RPCLogStreamProcessor.process(rpcStream, taskConfBroadcast)
//HTTP Log Streaming
val httpStream = KafkaUtils.createStream[String, String, StringDecoder,
StringDecoder](ssc, consumerParams, topicHTTP, StorageLevel.MEMORY_ONLY)
HttpLogStreamProcessor.process(httpStream, taskConfBroadcast)
There is a log information showing on the console in red color
[Stage 0:> (0 + 2) / 2]
It appears, then disappear, and then appear, disappear...
For the above code, if I only have rpc streaming and comment the httpStream,
then it disappear. I don't know how it occurs and how to suppress it
bit1...@163.com
