Re: mapWithState and DataFrames

2016-11-06 Thread Victor Shafran
=> rdd.toDF("word", "count").registerTempTable("batch_word_count") } Hope it helps Victor On Sun, Nov 6, 2016 at 12:53 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > How can I utilize mapWithState and DataFrames? > Rig

mapWithState and DataFrames

2016-11-06 Thread Daniel Haviv
Hi, How can I utilize mapWithState and DataFrames? Right now I stream json messages from Kafka, update their state, output the updated state as json and compose a dataframe from it. It seems inefficient both in terms of processing and storage (a long string instead of a compact DF