I have nested structure which i read from an xml using spark-Xml. I want to
use spark sql to convert this nested structure to different relational
tables
(WrappedArray([WrappedArray([[null,592006340,null],null,BA,M,1724]),N,2017-04-05T16:31:03,586257528),659925562)
which has a schema:
StructType(StructField(AirSegment,ArrayType(StructType(
StructField(CodeshareDetails,ArrayType(StructType(StructField(Links,StructType(StructField(_VALUE,StringType,true),
StructField(_mktSegmentID,LongType,true),
StructField(_oprSegmentID,LongType,true)),true),
StructField(_alphaSuffix,StringType,true),
StructField(_carrierCode,StringType,true),
StructField(_codeshareType,StringType,true),
StructField(_flightNumber,StringType,true)),true),true),
StructField(_adsIsDeleted,StringType,true),
StructField(_adsLastUpdateTimestamp,StringType,true),
StructField(_AirID,LongType,true)),true),true),
StructField(flightId,LongType,true))
*Question: Now as you can see this codeshareDetails is a wrappedArray inside
a Wrapped array. How can I extract these wrapped array rows along with the
_AirID column so that I can insert these rows in the codeshare table
(sqliteDb) (having column related to codeshare only along with _AirID as
foreign key, used for joining back).*
*PS:I tried exploding but in case if there are multiple rows in the
AirSegment array it doesnt work properly*
My table Structure is mentioned below:
Flight-contatining flightId and other Details:
AirSegment: Containing _AirID(PK), flightID(FK), and AirSegmentDetails
CodeshareDetails: containing CodeshareDetails as well as _AirID(FK)
Let me know if you need any more information
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/WrappedArray-to-row-of-relational-Db-tp28625.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
