This almost had me not using Spark; I couldn't get any output. It is not
at all obvious what's going on here to the layman (and to the best of my
knowledge, not documented anywhere), but now you know you'll be able to
answer this question for the numerous people that will also have it.
On Sun, Jul 13, 2014 at 5:24 PM, Walrus theCat <walrusthe...@gmail.com>
wrote:
> Great success!
>
> I was able to get output to the driver console by changing the
> construction of the Streaming Spark Context from:
>
> val ssc = new StreamingContext("local" /**TODO change once a cluster is
> up **/,
> "AppName", Seconds(1))
>
>
> to:
>
> val ssc = new StreamingContext("local[2]" /**TODO change once a cluster is
> up **/,
> "AppName", Seconds(1))
>
>
> I found something that tipped me off that this might work by digging
> through this mailing list.
>
>
> On Sun, Jul 13, 2014 at 2:00 PM, Walrus theCat <walrusthe...@gmail.com>
> wrote:
>
>> More strange behavior:
>>
>> lines.foreachRDD(x => println(x.first)) // works
>> lines.foreachRDD(x => println((x.count,x.first))) // no output is printed
>> to driver console
>>
>>
>>
>>
>> On Sun, Jul 13, 2014 at 11:47 AM, Walrus theCat <walrusthe...@gmail.com>
>> wrote:
>>
>>>
>>> Thanks for your interest.
>>>
>>> lines.foreachRDD(x => println(x.count))
>>>
>>> And I got 0 every once in a while (which I think is strange, because
>>> lines.print prints the input I'm giving it over the socket.)
>>>
>>>
>>> When I tried:
>>>
>>> lines.map(_->1).reduceByKey(_+_).foreachRDD(x => println(x.count))
>>>
>>> I got no count.
>>>
>>> Thanks
>>>
>>>
>>> On Sun, Jul 13, 2014 at 11:34 AM, Tathagata Das <
>>> tathagata.das1...@gmail.com> wrote:
>>>
>>>> Try doing DStream.foreachRDD and then printing the RDD count and
>>>> further inspecting the RDD.
>>>> On Jul 13, 2014 1:03 AM, "Walrus theCat" <walrusthe...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a DStream that works just fine when I say:
>>>>>
>>>>> dstream.print
>>>>>
>>>>> If I say:
>>>>>
>>>>> dstream.map(_,1).print
>>>>>
>>>>> that works, too. However, if I do the following:
>>>>>
>>>>> dstream.reduce{case(x,y) => x}.print
>>>>>
>>>>> I don't get anything on my console. What's going on?
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>
