[jira] [Updated] (SPARK-3833) Allow Spark SQL SchemaRDDs to be merged
[ https://issues.apache.org/jira/browse/SPARK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-3833: - Assignee: Michael Armbrust > Allow Spark SQL SchemaRDDs to be merged > --- > > Key: SPARK-3833 > URL: https://issues.apache.org/jira/browse/SPARK-3833 > Project: Spark > Issue Type: Wish > Components: SQL >Reporter: Chris Wood >Assignee: Michael Armbrust > Fix For: 1.4.0 > > > We have JSON flowing into Spark SQL. > I can successfully store them as parquet and read them with > sqlContext.jsonRDD, but the inferred schemas cannot be merged into a single > table to do queries. > I'd like a way to allow for parquet file schemas to be merged, whether they > match or not, since we know the schema should be a union of the schemas from > the files. > This will allow us to have the data define the schema and new columns will > just appear. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3833) Allow Spark SQL SchemaRDDs to be merged
[ https://issues.apache.org/jira/browse/SPARK-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Wood updated SPARK-3833: -- Summary: Allow Spark SQL SchemaRDDs to be merged (was: Allow Spark SQL SchemaRDDs to me berged) Allow Spark SQL SchemaRDDs to be merged --- Key: SPARK-3833 URL: https://issues.apache.org/jira/browse/SPARK-3833 Project: Spark Issue Type: Wish Components: SQL Reporter: Chris Wood We have JSON flowing into Spark SQL. I can successfully store them as parquet and read them with sqlContext.jsonRDD, but the inferred schemas cannot be merged into a single table to do queries. I'd like a way to allow for parquet file schemas to be merged, whether they match or not, since we know the schema should be a union of the schemas from the files. This will allow us to have the data define the schema and new columns will just appear. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
