[ https://issues.apache.org/jira/browse/SPARK-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian resolved SPARK-6607. ------------------------------- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5263 [https://github.com/apache/spark/pull/5263] > Aggregation attribute name including special chars '(' and ')' should be > replaced before generating Parquet schema > ------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-6607 > URL: https://issues.apache.org/jira/browse/SPARK-6607 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.1.1, 1.2.1, 1.3.0 > Reporter: Liang-Chi Hsieh > Assignee: Liang-Chi Hsieh > Fix For: 1.4.0 > > > '(' and ')' are special characters used in Parquet schema for type > annotation. When we run an aggregation query, we will obtain attribute name > such as "MAX(a)". > If we directly store the generated DataFrame as Parquet file, it causes > failure when reading and parsing the stored schema string. > Several methods can be adopted to solve this. This pr uses a simplest one to > just replace attribute names before generating Parquet schema based on these > attributes. > Another possible method might be modifying all aggregation expression names > from "func(column)" to "func[column]". -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org