This is not normal. Its a huge scheduling delay!! Can you tell me more about the application? - cluser setup, number of receivers, whats the computation, etc.
On Thu, Jan 22, 2015 at 3:11 AM, Ashic Mahtab <as...@live.com> wrote: > Hate to do this...but...erm...bump? Would really appreciate input from > others using Streaming. Or at least some docs that would tell me if these > are expected or not. > > ------------------------------ > From: as...@live.com > To: user@spark.apache.org > Subject: Are these numbers abnormal for spark streaming? > Date: Wed, 21 Jan 2015 11:26:31 +0000 > > > Hi Guys, > I've got Spark Streaming set up for a low data rate system (using spark's > features for analysis, rather than high throughput). Messages are coming in > throughout the day, at around 1-20 per second (finger in the air > estimate...not analysed yet). In the spark streaming UI for the > application, I'm getting the following after 17 hours. > > Streaming > > - *Started at: *Tue Jan 20 16:58:43 GMT 2015 > - *Time since start: *18 hours 24 minutes 34 seconds > - *Network receivers: *2 > - *Batch interval: *2 seconds > - *Processed batches: *16482 > - *Waiting batches: *1 > > > > Statistics over last 100 processed batchesReceiver Statistics > > - Receiver > > > - Status > > > - Location > > > - Records in last batch > - [2015/01/21 11:23:18] > > > - Minimum rate > - [records/sec] > > > - Median rate > - [records/sec] > > > - Maximum rate > - [records/sec] > > > - Last Error > > RmqReceiver-0ACTIVEFOOOO > 144727-RmqReceiver-1ACTIVEBAAAAR > 124726- > Batch Processing Statistics > > MetricLast batchMinimum25th percentileMedian75th > percentileMaximumProcessing > Time3 seconds 994 ms157 ms4 seconds 16 ms4 seconds 961 ms5 seconds 3 ms5 > seconds 171 msScheduling Delay9 hours 15 minutes 4 seconds9 hours 10 > minutes 54 seconds9 hours 11 minutes 56 seconds9 hours 12 minutes 57 > seconds9 hours 14 minutes 5 seconds9 hours 15 minutes 4 secondsTotal > Delay9 hours 15 minutes 8 seconds9 hours 10 minutes 58 seconds9 hours > 12 minutes9 hours 13 minutes 2 seconds9 hours 14 minutes 10 seconds9 > hours 15 minutes 8 seconds > > > Are these "normal". I was wondering what the scheduling delay and total > delay terms are, and if it's normal for them to be 9 hours. > > I've got a standalone spark master and 4 spark nodes. The streaming app > has been given 4 cores, and it's using 1 core per worker node. The > streaming app is submitted from a 5th machine, and that machine has nothing > but the driver running. The worker nodes are running alongside Cassandra > (and reading and writing to it). > > Any insights would be appreciated. > > Regards, > Ashic. >
