...bump? Would really appreciate input from
others using Streaming. Or at least some docs that would tell me if these
are expected or not.
--
From: as...@live.com
To: user@spark.apache.org
Subject: Are these numbers abnormal for spark streaming?
Date: Wed, 21 Jan 2015
:30:08 +0100
Subject: Re: Are these numbers abnormal for spark streaming?
To: tathagata.das1...@gmail.com
CC: as...@live.com; t...@databricks.com; user@spark.apache.org
and post the code (if possible).In a nutshell, your processing time batch
interval, resulting in an ever-increasing delay
from
others using Streaming. Or at least some docs that would tell me if these
are expected or not.
--
From: as...@live.com
To: user@spark.apache.org
Subject: Are these numbers abnormal for spark streaming?
Date: Wed, 21 Jan 2015 11:26:31 +
Hi Guys,
I've got
Hate to do this...but...erm...bump? Would really appreciate input from others
using Streaming. Or at least some docs that would tell me if these are expected
or not.
From: as...@live.com
To: user@spark.apache.org
Subject: Are these numbers abnormal for spark streaming?
Date: Wed, 21 Jan 2015 11
-20150120165844-0005App1
4500.0 MB2015/01/20 16:58:44rootWAITING42.4 h
From: tathagata.das1...@gmail.com
Date: Thu, 22 Jan 2015 03:15:58 -0800
Subject: Re: Are these numbers abnormal for spark streaming?
To: as...@live.com; t...@databricks.com
CC: user@spark.apache.org
This is not normal. Its a huge scheduling
ms4 seconds 701 ms4 seconds 739 ms4 seconds 764 ms4
seconds 792 ms5 seconds 809 ms
Regards,
Ashic.
--
From: as...@live.com
To: gerard.m...@gmail.com
CC: user@spark.apache.org
Subject: RE: Are these numbers abnormal for spark streaming?
Date: Thu, 22 Jan 2015
machines to be
involved (even better if this can be specified during spark-submit)?
Thanks,
Ashic.
From: as...@live.com
To: gerard.m...@gmail.com; asudipta.baner...@gmail.com
CC: user@spark.apache.org; tathagata.das1...@gmail.com
Subject: RE: Are these numbers abnormal for spark streaming?
Date: Thu
Hi Sudipta,
Standalone spark master. Separate Zookeeper cluster. 4 worker nodes with
cassandra + spark on each. No hadoop / hdfs / yarn.
Regards,
Ashic.
Date: Thu, 22 Jan 2015 20:42:43 +0530
Subject: Re: Are these numbers abnormal for spark streaming?
From: asudipta.baner...@gmail.com
To: gerard.m...@gmail.com
CC: user@spark.apache.org
Subject: RE: Are these numbers abnormal for spark streaming?
Date: Thu, 22 Jan 2015 12:32:05 +
Hi Gerard,
Thanks for the response.
The messages get desrialised from msgpack format, and one of the strings
is desrialised to json. Certain
...@gmail.com
CC: user@spark.apache.org
Subject: RE: Are these numbers abnormal for spark streaming?
Date: Thu, 22 Jan 2015 12:32:05 +
Hi Gerard,
Thanks for the response.
The messages get desrialised from msgpack format, and one of the strings is
desrialised to json. Certain fields are checked
seconds 701 ms4 seconds 739 ms4 seconds 764 ms4
seconds 792 ms5 seconds 809 ms
Regards,
Ashic.
--
From: as...@live.com
To: gerard.m...@gmail.com
CC: user@spark.apache.org
Subject: RE: Are these numbers abnormal for spark streaming?
Date: Thu, 22 Jan 2015 12
Yup...looks like it. I can do some tricks to reduce setup costs further, but
this is much better than where I was yesterday. Thanks for your awesome input :)
-Ashic.
From: gerard.m...@gmail.com
Date: Thu, 22 Jan 2015 16:34:38 +0100
Subject: Re: Are these numbers abnormal for spark streaming
Hi Guys,
I've got Spark Streaming set up for a low data rate system (using spark's
features for analysis, rather than high throughput). Messages are coming in
throughout the day, at around 1-20 per second (finger in the air estimate...not
analysed yet). In the spark streaming UI for the
13 matches
Mail list logo
