[ https://issues.apache.org/jira/browse/SPARK-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-15804. --------------------------------- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13555 [https://github.com/apache/spark/pull/13555] > Manually added metadata not saving with parquet > ----------------------------------------------- > > Key: SPARK-15804 > URL: https://issues.apache.org/jira/browse/SPARK-15804 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Reporter: Charlie Evans > Fix For: 2.0.0 > > > Adding metadata with col().as(_, metadata) then saving the resultant > dataframe does not save the metadata. No error is thrown. Only see the schema > contains the metadata before saving and does not contain the metadata after > saving and loading the dataframe. Was working fine with 1.6.1. > {code} > case class TestRow(a: String, b: Int) > val rows = TestRow("a", 0) :: TestRow("b", 1) :: TestRow("c", 2) :: Nil > val df = spark.createDataFrame(rows) > import org.apache.spark.sql.types.MetadataBuilder > val md = new MetadataBuilder().putString("key", "value").build() > val dfWithMeta = df.select(col("a"), col("b").as("b", md)) > println(dfWithMeta.schema.json) > dfWithMeta.write.parquet("dfWithMeta") > val dfWithMeta2 = spark.read.parquet("dfWithMeta") > println(dfWithMeta2.schema.json) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org