[
https://issues.apache.org/jira/browse/SPARK-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530971#comment-15530971
]
Dennis Hunziker edited comment on SPARK-9139 at 9/28/16 9:47 PM:
-----------------------------------------------------------------
[~dongjoon], the commit you mentioned is a change in documentation only. It
seems related to how the data types are inferred when reading data from JSON or
the docs were just slightly wrong to begin with.
For the issue raised here, I did compare the JSON representation of all the
available SQL data types and a few test schemas using versions from 1.3.1
onward and couldn't find any differences. I agree that for testing backwards
compatibility, the JSON strings should be saved down now and tested against the
current version of the code. If agreed I can pick this up.
was (Author: [email protected]):
@Dongjoon Hyun, the commit you mentioned is a change in documentation only. It
seems related to how the data types are inferred when reading data from JSON or
the docs were just slightly wrong to begin with.
For the issue raised here, I did compare the JSON representation of all the
available SQL data types and a few test schemas using versions from 1.3.1
onward and couldn't find any differences. I agree that for testing backwards
compatibility, the JSON strings should be saved down now and tested against the
current version of the code. If agreed I can pick this up.
> Add backwards-compatibility tests for DataType.fromJson()
> ---------------------------------------------------------
>
> Key: SPARK-9139
> URL: https://issues.apache.org/jira/browse/SPARK-9139
> Project: Spark
> Issue Type: Test
> Components: SQL
> Reporter: Josh Rosen
> Priority: Critical
>
> SQL's DataType.fromJson is a public API and thus must be
> backwards-compatible; there are also backwards-compatibility concerns related
> to persistence of DataType JSON in metastores.
> Unfortunately, we do not have any backwards-compatibility tests which attempt
> to read old JSON values that were written by earlier versions of Spark.
> DataTypeSuite has "roundtrip" tests that test fromJson(toJson(foo)), but this
> doesn't ensure compatibility.
> I think that we should address this by capuring the JSON strings produced in
> Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes
> from those strings.
> This might be a good starter task for someone who wants to contribute to SQL
> tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
