Thank you Jerry,
Does the window operation create new RDDs for each slide duration..?
I am asking this because i see a constant increase in memory even when
there is no logs received.
If not checkpoint is there any alternative that you would suggest.?
On Tue, Jan 20, 2015 at 7:08 PM, Shao, Saisai <saisai.s...@intel.com> wrote:
> Hi,
>
>
>
> Seems you have such a large window (24 hours), so the phenomena of memory
> increasing is expectable, because of window operation will cache the RDD
> within this window in memory. So for your requirement, memory should be
> enough to hold the data of 24 hours.
>
>
>
> I don’t think checkpoint in Spark Streaming can alleviate such problem,
> because checkpoint are mainly for fault tolerance.
>
>
>
> Thanks
>
> Jerry
>
>
>
> *From:* balu.naren [mailto:balu.na...@gmail.com]
> *Sent:* Tuesday, January 20, 2015 7:17 PM
> *To:* user@spark.apache.org
> *Subject:* spark streaming with checkpoint
>
>
>
> I am a beginner to spark streaming. So have a basic doubt regarding
> checkpoints. My use case is to calculate the no of unique users by day. I
> am using reduce by key and window for this. Where my window duration is 24
> hours and slide duration is 5 mins. I am updating the processed record to
> mongodb. Currently I am replace the existing record each time. But I see
> the memory is slowly increasing over time and kills the process after 1 and
> 1/2 hours(in aws small instance). The DB write after the restart clears all
> the old data. So I understand checkpoint is the solution for this. But my
> doubt is
>
> - What should my check point duration be..? As per documentation it
> says 5-10 times of slide duration. But I need the data of entire day. So it
> is ok to keep 24 hrs.
> - Where ideally should the checkpoint be..? Initially when I receive
> the stream or just before the window operation or after the data reduction
> has taken place.
>
>
> Appreciate your help.
> Thank you
> ------------------------------
>
> View this message in context: spark streaming with checkpoint
> <http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-with-checkpoint-tp21263.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
