[jira] [Updated] (SPARK-4523) Improve handling of serialized schema information
[ https://issues.apache.org/jira/browse/SPARK-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4523: Priority: Critical (was: Blocker) > Improve handling of serialized schema information > - > > Key: SPARK-4523 > URL: https://issues.apache.org/jira/browse/SPARK-4523 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Michael Armbrust >Priority: Critical > > There are several issues with our current handling of metadata serialization, > which is especially troublesome since this is the only place that we persist > information directly using Spark SQL. Moving forward we should do the > following: > - Relax the parsing so that it does not fail when optional fields are > missing (i.e. containsNull or metadata) > - Include a regression suite that attempts to read old parquet files written > by previous versions of Spark SQL. > - Provide better warning messages when various forms of parsing fail (I > think that it is silent right now which makes tracking down bugs more > difficult than it needs to be). > - Deprecate (display a warning) when reading data with the old case class > schema representation and eventually remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-4523) Improve handling of serialized schema information
[ https://issues.apache.org/jira/browse/SPARK-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4523: Target Version/s: 1.4.0 (was: 1.3.0) > Improve handling of serialized schema information > - > > Key: SPARK-4523 > URL: https://issues.apache.org/jira/browse/SPARK-4523 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Michael Armbrust >Priority: Critical > > There are several issues with our current handling of metadata serialization, > which is especially troublesome since this is the only place that we persist > information directly using Spark SQL. Moving forward we should do the > following: > - Relax the parsing so that it does not fail when optional fields are > missing (i.e. containsNull or metadata) > - Include a regression suite that attempts to read old parquet files written > by previous versions of Spark SQL. > - Provide better warning messages when various forms of parsing fail (I > think that it is silent right now which makes tracking down bugs more > difficult than it needs to be). > - Deprecate (display a warning) when reading data with the old case class > schema representation and eventually remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
