Re: Help with Spark Streaming

2014-11-16 Thread Bahubali Jain
Hi, Can anybody help me on this please, haven't been able to find the problem :( Thanks. On Nov 15, 2014 4:48 PM, Bahubali Jain bahub...@gmail.com wrote: Hi, Trying to use spark streaming, but I am struggling with word count :( I want consolidate output of the word count (not on a per window

Re: Help with Spark Streaming

2014-11-16 Thread ZhangYi
I guess, maybe you don’t need invoke reduceByKey() after mapToPair, because updateStateByKey had covered it. For your reference, here is a sample written by scala using text file stream instead of socket as below: object LocalStatefulWordCount extends App { val sparkConf = new

Help with Spark Streaming

2014-11-15 Thread Bahubali Jain
Hi, Trying to use spark streaming, but I am struggling with word count :( I want consolidate output of the word count (not on a per window basis), so I am using updateStateByKey(), but for some reason this is not working. The function it self is not being invoked(do not see the sysout output on