Re: JSON to SQL

2016-01-28 Thread Andrés Ivaldi
gt; <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> > > > > *From:* Andrés Ivaldi [mailto:iaiva...@gmail.com] > *Sent:* Wednesday, January 27, 2016 7:17 PM > *To:* Cheng, Hao > *Cc:* Sahil Sareen; Al Pivonka; user > > *Subject:* Re: JS

RE: JSON to SQL

2016-01-28 Thread Mohammed Guller
day, January 27, 2016 7:17 PM To: Cheng, Hao Cc: Sahil Sareen; Al Pivonka; user Subject: Re: JSON to SQL I'm using DataFrames reading the JSON exactly as you say, and I can get the scheme from there. Reading the documentation, I realized that is possible to create Dynamically a Stru

RE: JSON to SQL

2016-01-27 Thread Cheng, Hao
“a.b[0].c” format of expression. From: Andrés Ivaldi [mailto:iaiva...@gmail.com] Sent: Thursday, January 28, 2016 3:39 AM To: Sahil Sareen Cc: Al Pivonka; user Subject: Re: JSON to SQL I'm really brand new with Scala, but if I'm defining a case class then is becouse I know how is the json's

Re: JSON to SQL

2016-01-27 Thread Andrés Ivaldi
/Hive/LanguageManual+LateralView, > as well as the “a.b[0].c” format of expression. > > > > > > *From:* Andrés Ivaldi [mailto:iaiva...@gmail.com] > *Sent:* Thursday, January 28, 2016 3:39 AM > *To:* Sahil Sareen > *Cc:* Al Pivonka; user > *Subject:* Re: JSON to S

Re: JSON to SQL

2016-01-27 Thread Al Pivonka
Are you using an Relational Database? If so why not use a nojs DB ? then pull from it to your relational? Or utilize a library that understands Json structure like Jackson to obtain the data from the Json structure the persist the Domain Objects ? On Wed, Jan 27, 2016 at 9:45 AM, Andrés Ivaldi

Re: JSON to SQL

2016-01-27 Thread Andrés Ivaldi
We dont have Domain Objects, its a service like a pipeline, data is read from source and they are saved it in relational Database I can read the structure from DataFrames, and do some transformations, I would prefer to do it with Spark to be consistent with the process On Wed, Jan 27, 2016 at

Re: JSON to SQL

2016-01-27 Thread Sahil Sareen
Isn't this just about defining a case class and using parse(json).extract[CaseClassName] using Jackson? -Sahil On Wed, Jan 27, 2016 at 11:08 PM, Andrés Ivaldi wrote: > We dont have Domain Objects, its a service like a pipeline, data is read > from source and they are saved

Re: JSON to SQL

2016-01-27 Thread Andrés Ivaldi
I'm really brand new with Scala, but if I'm defining a case class then is becouse I know how is the json's structure is previously? If I'm able to define dinamicaly a case class from the JSON structure then even with spark I will be able to extract the data On Wed, Jan 27, 2016 at 4:01 PM,

Re: JSON to SQL

2016-01-27 Thread Al Pivonka
More detail is needed. Can you provide some context to the use-case ? On Wed, Jan 27, 2016 at 8:33 AM, Andrés Ivaldi wrote: > Hello, I'm trying to Save a JSON filo into SQL table. > > If i try to do this directly the IlligalArgumentException is raised, I > suppose this is

Re: JSON to SQL

2016-01-27 Thread Andrés Ivaldi
Sure, The Job is like an etl, but without interface, so I decide the rules of how the JSON will be saved into a SQL Table. I need to Flatten the hierarchies where is possible in case of list flatten also, nested objects Won't be processed by now {"a":1,"b":[2,3],"c"="Field", "d":[4,5,6,7,8] }