[
https://issues.apache.org/jira/browse/SPARK-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-4523:
------------------------------------
Priority: Critical (was: Blocker)
> Improve handling of serialized schema information
> -------------------------------------------------
>
> Key: SPARK-4523
> URL: https://issues.apache.org/jira/browse/SPARK-4523
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Michael Armbrust
> Priority: Critical
>
> There are several issues with our current handling of metadata serialization,
> which is especially troublesome since this is the only place that we persist
> information directly using Spark SQL. Moving forward we should do the
> following:
> - Relax the parsing so that it does not fail when optional fields are
> missing (i.e. containsNull or metadata)
> - Include a regression suite that attempts to read old parquet files written
> by previous versions of Spark SQL.
> - Provide better warning messages when various forms of parsing fail (I
> think that it is silent right now which makes tracking down bugs more
> difficult than it needs to be).
> - Deprecate (display a warning) when reading data with the old case class
> schema representation and eventually remove it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
