Re: spark streaming with checkpoint

2015-01-25 Thread Balakrishnan Narendran
- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: spark streaming with checkpoint

2015-01-25 Thread Balakrishnan Narendran
Yeah use streaming to gather the incoming logs and write to log file then run a spark job evry 5 minutes to process the counts. Got it. Thanks a lot. On 07:07, Mon, 26 Jan 2015 Tobias Pfeiffer wrote: > Hi, > > On Tue, Jan 20, 2015 at 8:16 PM, balu.naren wrote: > >> I am a beginner to spark

Re: spark streaming with checkpoint

2015-01-25 Thread Tobias Pfeiffer
Hi, On Tue, Jan 20, 2015 at 8:16 PM, balu.naren wrote: > I am a beginner to spark streaming. So have a basic doubt regarding > checkpoints. My use case is to calculate the no of unique users by day. I > am using reduce by key and window for this. Where my window duration is 24 > hours and slide

RE: spark streaming with checkpoint

2015-01-22 Thread Shao, Saisai
PM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: spark streaming with checkpoint I am a beginner to spark streaming. So have a basic doubt regarding checkpoints. My use case is to calculate the no of unique users by day. I am using reduce by key and window for thi

Re: spark streaming with checkpoint

2015-01-22 Thread Jörn Franke
o hold the data of 24 hours. >> >> >> >> I don’t think checkpoint in Spark Streaming can alleviate such problem, >> because checkpoint are mainly for fault tolerance. >> >> >> >> Thanks >> >> Jerry >> >> >> >> *From:

Re: spark streaming with checkpoint

2015-01-22 Thread Balakrishnan Narendran
esday, January 20, 2015 7:17 PM > *To:* user@spark.apache.org > *Subject:* spark streaming with checkpoint > > > > I am a beginner to spark streaming. So have a basic doubt regarding > checkpoints. My use case is to calculate the no of unique users by day. I > am using reduce by ke

RE: spark streaming with checkpoint

2015-01-20 Thread Shao, Saisai
Spark Streaming can alleviate such problem, because checkpoint are mainly for fault tolerance. Thanks Jerry From: balu.naren [mailto:balu.na...@gmail.com] Sent: Tuesday, January 20, 2015 7:17 PM To: user@spark.apache.org Subject: spark streaming with checkpoint I am a beginner to spark streamin

spark streaming with checkpoint

2015-01-20 Thread balu.naren
reduction has taken place. Appreciate your help. Thank you -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-with-checkpoint-tp21263.html Sent from the Apache Spark User List mailing list archive at Nabble.com.