Re: parsing embedded json in spark

Hi Shaw, Thanks, that works! On Thu, Dec 22, 2016 at 6:45 PM, Shaw Liu wrote: > Hi,I guess you can use 'get_json_object' function > > Get Outlook for iOS > > > > > On Thu, Dec 22, 2016 at 9:52 PM +0800, "Irving Duran" < > irving.du...@gmail.com> wrote: > > Is it an opt

Re: parsing embedded json in spark

Hi,I guess you can use 'get_json_object' function Get Outlook for iOS On Thu, Dec 22, 2016 at 9:52 PM +0800, "Irving Duran" wrote: Is it an option to parse that field prior of creating the dataframe? If so, that's what I would do. In terms of your master node only working, you hav

Re: parsing embedded json in spark

Is it an option to parse that field prior of creating the dataframe? If so, that's what I would do. In terms of your master node only working, you have to share more about your structure, are you using spark standalone, yarn, or mesos? Thank You, Irving Duran On Thu, Dec 22, 2016 at 1:42 AM, T

parsing embedded json in spark

Hi, I have a dataframe that contain an embedded json string in one of the fields I'd tried to write a UDF function that will parse it using lift-json, but it seems to take a very long time to process, and it seems that only the master node is working. Has anyone dealt with such a scenario before