Re: How to convert Dstream of JsonObject to Dataframe in spark 2.1.0?

[kant kodali]

Thanks sam!


On Mon, Apr 24, 2017 at 1:50 AM, Sam Elamin  wrote:

> you have 2 options
> 1 )Clean ->Write your own parser to through each property and create a
> dataset
> 2) Hacky but simple -> Convert to json string then read in using
> spark.read.json(jsonString)
>
> Please bear in mind the second option is expensive which is why it is hacky
>
> I wrote my own parser here
> 
> which you can use to convert between JsonObjects to StructType schemas
>
> Regards
> Sam
>
>
> On Sun, Apr 23, 2017 at 7:50 PM, kant kodali  wrote:
>
>> Hi All,
>>
>> How to convert Dstream of JsonObject to Dataframe in spark 2.1.0? That
>> JsonObject is from Gson Library.
>>
>> Thanks!
>>
>
>

Re: How to convert Dstream of JsonObject to Dataframe in spark 2.1.0?

[Sam Elamin]

you have 2 options
1 )Clean ->Write your own parser to through each property and create a
dataset
2) Hacky but simple -> Convert to json string then read in using
spark.read.json(jsonString)

Please bear in mind the second option is expensive which is why it is hacky

I wrote my own parser here

which you can use to convert between JsonObjects to StructType schemas

Regards
Sam


On Sun, Apr 23, 2017 at 7:50 PM, kant kodali  wrote:

> Hi All,
>
> How to convert Dstream of JsonObject to Dataframe in spark 2.1.0? That
> JsonObject is from Gson Library.
>
> Thanks!
>

How to convert Dstream of JsonObject to Dataframe in spark 2.1.0?

[kant kodali]

Hi All,

How to convert Dstream of JsonObject to Dataframe in spark 2.1.0? That
JsonObject is from Gson Library.

Thanks!