Yeah use streaming to gather the incoming logs and write to log file then run a spark job evry 5 minutes to process the counts. Got it. Thanks a lot.!!!!
On 07:07, Mon, 26 Jan 2015 Tobias Pfeiffer <t...@preferred.jp> wrote: > Hi, > > On Tue, Jan 20, 2015 at 8:16 PM, balu.naren <balu.na...@gmail.com> wrote: > >> I am a beginner to spark streaming. So have a basic doubt regarding >> checkpoints. My use case is to calculate the no of unique users by day. I >> am using reduce by key and window for this. Where my window duration is 24 >> hours and slide duration is 5 mins. >> > Adding to what others said, this feels more like a task for "run a Spark > job every five minutes using cron" than using the sliding window > functionality from Spark Streaming. > > Tobias >
