Re: parsing embedded json in spark

[Tal Grynbaum]

Hi Shaw,

Thanks, that works!



On Thu, Dec 22, 2016 at 6:45 PM, Shaw Liu  wrote:

> Hi,I guess you can use 'get_json_object' function
>
> Get Outlook for iOS 
>
>
>
>
> On Thu, Dec 22, 2016 at 9:52 PM +0800, "Irving Duran" <
> irving.du...@gmail.com> wrote:
>
> Is it an option to parse that field prior of creating the dataframe? If
>> so, that's what I would do.
>>
>> In terms of your master node only working, you have to share more about
>> your structure, are you using spark standalone, yarn, or mesos?
>>
>>
>> Thank You,
>>
>> Irving Duran
>>
>> On Thu, Dec 22, 2016 at 1:42 AM, Tal Grynbaum 
>> wrote:
>>
>>> Hi,
>>>
>>> I have a dataframe that contain an embedded json string in one of the
>>> fields
>>> I'd tried to write a UDF function that will parse it using lift-json,
>>> but it seems to take a very long time to process, and it seems that only
>>> the master node is working.
>>>
>>> Has anyone dealt with such a scenario before and can give me some hints?
>>>
>>> Thanks
>>> Tal
>>>
>>
>>


-- 
*Tal Grynbaum* / *CTO & co-founder*

m# +972-54-7875797

mobile retention done right

Re: parsing embedded json in spark

[Shaw Liu]

Hi,I guess you can use 'get_json_object' function

Get Outlook for iOS




On Thu, Dec 22, 2016 at 9:52 PM +0800, "Irving Duran"  
wrote:










Is it an option to parse that field prior of creating the dataframe? If so, 
that's what I would do.
In terms of your master node only working, you have to share more about your 
structure, are you using spark standalone, yarn, or mesos?

Thank You,

Irving Duran

On Thu, Dec 22, 2016 at 1:42 AM, Tal Grynbaum  wrote:
Hi, 

I have a dataframe that contain an embedded json string in one of the fieldsI'd 
tried to write a UDF function that will parse it using lift-json, but it seems 
to take a very long time to process, and it seems that only the master node is 
working.
Has anyone dealt with such a scenario before and can give me some hints?
ThanksTal

Re: parsing embedded json in spark

[Irving Duran]

Is it an option to parse that field prior of creating the dataframe? If so,
that's what I would do.

In terms of your master node only working, you have to share more about
your structure, are you using spark standalone, yarn, or mesos?


Thank You,

Irving Duran

On Thu, Dec 22, 2016 at 1:42 AM, Tal Grynbaum 
wrote:

> Hi,
>
> I have a dataframe that contain an embedded json string in one of the
> fields
> I'd tried to write a UDF function that will parse it using lift-json, but
> it seems to take a very long time to process, and it seems that only the
> master node is working.
>
> Has anyone dealt with such a scenario before and can give me some hints?
>
> Thanks
> Tal
>

parsing embedded json in spark

[Tal Grynbaum]

Hi,

I have a dataframe that contain an embedded json string in one of the fields
I'd tried to write a UDF function that will parse it using lift-json, but
it seems to take a very long time to process, and it seems that only the
master node is working.

Has anyone dealt with such a scenario before and can give me some hints?

Thanks
Tal