Have you ever try the DataFrame API like:
sqlContext.read.json("/path/to/file.json"); the Spark SQL will auto infer the
type/schema for you.
And lateral view will help on the flatten issues,
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView, as
well as the “a.b[0].c” format of expression.
From: Andrés Ivaldi [mailto:iaiva...@gmail.com]
Sent: Thursday, January 28, 2016 3:39 AM
To: Sahil Sareen
Cc: Al Pivonka; user
Subject: Re: JSON to SQL
I'm really brand new with Scala, but if I'm defining a case class then is
becouse I know how is the json's structure is previously?
If I'm able to define dinamicaly a case class from the JSON structure then even
with spark I will be able to extract the data
On Wed, Jan 27, 2016 at 4:01 PM, Sahil Sareen
<sareen...@gmail.com<mailto:sareen...@gmail.com>> wrote:
Isn't this just about defining a case class and using
parse(json).extract[CaseClassName] using Jackson?
-Sahil
On Wed, Jan 27, 2016 at 11:08 PM, Andrés Ivaldi
<iaiva...@gmail.com<mailto:iaiva...@gmail.com>> wrote:
We dont have Domain Objects, its a service like a pipeline, data is read from
source and they are saved it in relational Database
I can read the structure from DataFrames, and do some transformations, I would
prefer to do it with Spark to be consistent with the process
On Wed, Jan 27, 2016 at 12:25 PM, Al Pivonka
<alpivo...@gmail.com<mailto:alpivo...@gmail.com>> wrote:
Are you using an Relational Database?
If so why not use a nojs DB ? then pull from it to your relational?
Or utilize a library that understands Json structure like Jackson to obtain the
data from the Json structure the persist the Domain Objects ?
On Wed, Jan 27, 2016 at 9:45 AM, Andrés Ivaldi
<iaiva...@gmail.com<mailto:iaiva...@gmail.com>> wrote:
Sure,
The Job is like an etl, but without interface, so I decide the rules of how the
JSON will be saved into a SQL Table.
I need to Flatten the hierarchies where is possible in case of list flatten
also, nested objects Won't be processed by now
{"a":1,"b":[2,3],"c"="Field", "d":[4,5,6,7,8] }
{"a":11,"b":[22,33],"c"="Field1", "d":[44,55,66,77,88] }
{"a":111,"b":[222,333],"c"="Field2", "d":[44,55,666,777,888] }
I would like something like this on my SQL table
a b c d
1 2,3 Field 4,5,6,7,8
11 22,33 Field1 44,55,66,77,88
111 222,333 Field2 444,555,,666,777,888
Right now this is what i need
I will later add more intelligence, like detection of list or nested objects
and create relations in other tables.
On Wed, Jan 27, 2016 at 11:25 AM, Al Pivonka
<alpivo...@gmail.com<mailto:alpivo...@gmail.com>> wrote:
More detail is needed.
Can you provide some context to the use-case ?
On Wed, Jan 27, 2016 at 8:33 AM, Andrés Ivaldi
<iaiva...@gmail.com<mailto:iaiva...@gmail.com>> wrote:
Hello, I'm trying to Save a JSON filo into SQL table.
If i try to do this directly the IlligalArgumentException is raised, I suppose
this is beacouse JSON have a hierarchical structure, is that correct?
If that is the problem, how can I flatten the JSON structure? The JSON
structure to be processed would be unknow, so I need to do it programatically
regards
--
Ing. Ivaldi Andres
--
Those who say it can't be done, are usually interrupted by those doing it.
--
Ing. Ivaldi Andres
--
Those who say it can't be done, are usually interrupted by those doing it.
--
Ing. Ivaldi Andres
--
Ing. Ivaldi Andres
