spark git commit: [MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes
Repository: spark Updated Branches: refs/heads/branch-2.3 5781fa79e -> 36af73b59 [MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes ## What changes were proposed in this pull request? This PR fixes the wrong comment on `org.apache.spark.sql.parquet.row.attributes` which is useful for UDTs like Vector/Matrix. Please see [SPARK-22320](https://issues.apache.org/jira/browse/SPARK-22320) for the usage. Originally, [SPARK-19411](https://github.com/apache/spark/commit/bf493686eb17006727b3ec81849b22f3df68fdef#diff-ee26d4c4be21e92e92a02e9f16dbc285L314) left this behind during removing optional column metadatas. In the same PR, the same comment was removed at line 310-311. ## How was this patch tested? N/A (This is about comments). Author: Dongjoon Hyun Closes #20346 from dongjoon-hyun/minor_comment_parquet. (cherry picked from commit 8142a3b883a5fe6fc620a2c5b25b6bde4fda32e5) Signed-off-by: hyukjinkwon Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/36af73b5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/36af73b5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/36af73b5 Branch: refs/heads/branch-2.3 Commit: 36af73b59b6fb3d5f8e8a8e1caf44bd565e97b3d Parents: 5781fa7 Author: Dongjoon Hyun Authored: Mon Jan 22 15:18:57 2018 +0900 Committer: hyukjinkwon Committed: Mon Jan 22 15:19:15 2018 +0900 -- .../sql/execution/datasources/parquet/ParquetFileFormat.scala | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/36af73b5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index 45bedf7..f53a97b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -108,8 +108,7 @@ class ParquetFileFormat ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport]) -// We want to clear this temporary metadata from saving into Parquet file. -// This metadata is only useful for detecting optional columns when pushdowning filters. +// This metadata is useful for keeping UDTs like Vector/Matrix. ParquetWriteSupport.setSchema(dataSchema, conf) // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes
Repository: spark Updated Branches: refs/heads/master 11daeb833 -> 8142a3b88 [MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes ## What changes were proposed in this pull request? This PR fixes the wrong comment on `org.apache.spark.sql.parquet.row.attributes` which is useful for UDTs like Vector/Matrix. Please see [SPARK-22320](https://issues.apache.org/jira/browse/SPARK-22320) for the usage. Originally, [SPARK-19411](https://github.com/apache/spark/commit/bf493686eb17006727b3ec81849b22f3df68fdef#diff-ee26d4c4be21e92e92a02e9f16dbc285L314) left this behind during removing optional column metadatas. In the same PR, the same comment was removed at line 310-311. ## How was this patch tested? N/A (This is about comments). Author: Dongjoon Hyun Closes #20346 from dongjoon-hyun/minor_comment_parquet. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8142a3b8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8142a3b8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8142a3b8 Branch: refs/heads/master Commit: 8142a3b883a5fe6fc620a2c5b25b6bde4fda32e5 Parents: 11daeb8 Author: Dongjoon Hyun Authored: Mon Jan 22 15:18:57 2018 +0900 Committer: hyukjinkwon Committed: Mon Jan 22 15:18:57 2018 +0900 -- .../sql/execution/datasources/parquet/ParquetFileFormat.scala | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/8142a3b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index 45bedf7..f53a97b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -108,8 +108,7 @@ class ParquetFileFormat ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport]) -// We want to clear this temporary metadata from saving into Parquet file. -// This metadata is only useful for detecting optional columns when pushdowning filters. +// This metadata is useful for keeping UDTs like Vector/Matrix. ParquetWriteSupport.setSchema(dataSchema, conf) // Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
