[
https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25102:
----------------------------------
Description:
Currently, Spark writes Spark version number into Hive Table properties with
`spark.sql.create.version`.
{code}
parameters:{
spark.sql.sources.schema.part.0={
"type":"struct",
"fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}}]
},
transient_lastDdlTime=1541142761,
spark.sql.sources.schema.numParts=1,
spark.sql.create.version=2.4.0
}
{code}
This issue aims to write Spark versions to ORC/Parquet file metadata with
`org.apache.spark.sql.create.version`. It's different from Hive Table property
key `spark.sql.create.version`. It seems that we cannot change that for
backward compatibility (even in Apache Spark 3.0)
*PARQUET*
{code}
file:
file:/tmp/p/part-00007-9dc415fe-7773-49ba-9c59-4c151e16009a-c000.snappy.parquet
creator: parquet-mr version 1.10.0 (build
031a6654009e3b82020012a18434c582bd74c73a)
extra: org.apache.spark.sql.create.version = 3.0.0-SNAPSHOT
extra: org.apache.spark.sql.parquet.row.metadata =
{"type":"struct","fields":[{"name":"id","type":"long","nullable":false,"metadata":{}}]}
{code}
was:
Currently, Spark writes Spark version number into Hive Table properties with
`spark.sql.create.version`.
{code}
parameters:{
spark.sql.sources.schema.part.0={
"type":"struct",
"fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}}]
},
transient_lastDdlTime=1541142761,
spark.sql.sources.schema.numParts=1,
spark.sql.create.version=2.4.0
}
{code}
This issue aims to write Spark versions to ORC/Parquet file metadata
consistently with `org.apache.spark.sql.create.version`.
*PARQUET*
{code}
file:
file:/tmp/p/part-00007-9dc415fe-7773-49ba-9c59-4c151e16009a-c000.snappy.parquet
creator: parquet-mr version 1.10.0 (build
031a6654009e3b82020012a18434c582bd74c73a)
extra: org.apache.spark.sql.create.version = 3.0.0-SNAPSHOT
extra: org.apache.spark.sql.parquet.row.metadata =
{"type":"struct","fields":[{"name":"id","type":"long","nullable":false,"metadata":{}}]}
{code}
> Write Spark version to ORC/Parquet file metadata
> ------------------------------------------------
>
> Key: SPARK-25102
> URL: https://issues.apache.org/jira/browse/SPARK-25102
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Zoltan Ivanfi
> Priority: Major
>
> Currently, Spark writes Spark version number into Hive Table properties with
> `spark.sql.create.version`.
> {code}
> parameters:{
> spark.sql.sources.schema.part.0={
> "type":"struct",
> "fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}}]
> },
> transient_lastDdlTime=1541142761,
> spark.sql.sources.schema.numParts=1,
> spark.sql.create.version=2.4.0
> }
> {code}
> This issue aims to write Spark versions to ORC/Parquet file metadata with
> `org.apache.spark.sql.create.version`. It's different from Hive Table
> property key `spark.sql.create.version`. It seems that we cannot change that
> for backward compatibility (even in Apache Spark 3.0)
> *PARQUET*
> {code}
> file:
> file:/tmp/p/part-00007-9dc415fe-7773-49ba-9c59-4c151e16009a-c000.snappy.parquet
> creator: parquet-mr version 1.10.0 (build
> 031a6654009e3b82020012a18434c582bd74c73a)
> extra: org.apache.spark.sql.create.version = 3.0.0-SNAPSHOT
> extra: org.apache.spark.sql.parquet.row.metadata =
> {"type":"struct","fields":[{"name":"id","type":"long","nullable":false,"metadata":{}}]}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
