Is it an option to parse that field prior of creating the dataframe? If so, that's what I would do.
In terms of your master node only working, you have to share more about your structure, are you using spark standalone, yarn, or mesos? Thank You, Irving Duran On Thu, Dec 22, 2016 at 1:42 AM, Tal Grynbaum <tal.grynb...@gmail.com> wrote: > Hi, > > I have a dataframe that contain an embedded json string in one of the > fields > I'd tried to write a UDF function that will parse it using lift-json, but > it seems to take a very long time to process, and it seems that only the > master node is working. > > Has anyone dealt with such a scenario before and can give me some hints? > > Thanks > Tal >
