[
https://issues.apache.org/jira/browse/SPARK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust resolved SPARK-3833.
-------------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
We support Schema merging for parquet starting with Spark 1.4.
> Allow Spark SQL SchemaRDDs to be merged
> ---------------------------------------
>
> Key: SPARK-3833
> URL: https://issues.apache.org/jira/browse/SPARK-3833
> Project: Spark
> Issue Type: Wish
> Components: SQL
> Reporter: Chris Wood
> Fix For: 1.4.0
>
>
> We have JSON flowing into Spark SQL.
> I can successfully store them as parquet and read them with
> sqlContext.jsonRDD, but the inferred schemas cannot be merged into a single
> table to do queries.
> I'd like a way to allow for parquet file schemas to be merged, whether they
> match or not, since we know the schema should be a union of the schemas from
> the files.
> This will allow us to have the data define the schema and new columns will
> just appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
