GitHub user fjh100456 opened a pull request:
https://github.com/apache/spark/pull/20087
[SPARK-21786][SQL] The 'spark.sql.parquet.compression.codec' and
'spark.sql.orc.compression.codec' configuration doesn't take effect on hive
table writing
[SPARK-21786][SQL] The 'spark.sql.parquet.compression.codec' and
'spark.sql.orc.compression.codec' configuration doesn't take effect on hive
table writing
What changes were proposed in this pull request?
Pass âspark.sql.parquet.compression.codecâ value to
âparquet.compressionâ.
Pass âspark.sql.orc.compression.codecâ value to âorc.compressâ.
How was this patch tested?
Add test.
Note:
This is the same issue mentioned in #19218 . That branch was deleted
mistakenly, so make a new pr instead.
@gatorsmile @maropu @dongjoon-hyun @discipleforteen
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fjh100456/spark HiveTableWriting
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20087.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20087
----
commit 9bbfe6ef4b5a418373c2250ad676233fb05df7f7
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T02:29:53Z
[SPARK-21786][SQL] When acquiring 'compressionCodecClassName' in
'ParquetOptions', `parquet.compression` needs to be considered.
## What changes were proposed in this pull request?
1.Increased acquiring 'compressionCodecClassName' from
`parquet.compression`,and the order is
`compression`,`parquet.compression`,`spark.sql.parquet.compression.codec`, just
like what we do in `OrcOptions`.
2.Change `spark.sql.parquet.compression.codec` to support "none".Actually
in `ParquetOptions`,we do support "none" as equivalent to "uncompressed", but
it does not allowed to configured to "none".
## How was this patch tested?
Manual test.
commit 48cf108ed5c3298eb860d9735b439ac89d65765e
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T02:30:24Z
[SPARK-21786][SQL] When acquiring 'compressionCodecClassName' in
'ParquetOptions', `parquet.compression` needs to be considered.
## What changes were proposed in this pull request?
1.Increased acquiring 'compressionCodecClassName' from
`parquet.compression`,and the order is
`compression`,`parquet.compression`,`spark.sql.parquet.compression.codec`, just
like what we do in `OrcOptions`.
2.Change `spark.sql.parquet.compression.codec` to support "none".Actually
in `ParquetOptions`,we do support "none" as equivalent to "uncompressed", but
it does not allowed to configured to "none".
## How was this patch tested?
Manual test.
commit 5dbd3edf9e086433d3d3fe9c0ead887d799c61d3
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T02:34:29Z
spark.sql.parquet.compression.codec[SPARK-21786][SQL] When acquiring
'compressionCodecClassName' in 'ParquetOptions', `parquet.compression` needs to
be considered.
## What changes were proposed in this pull request?
1.Increased acquiring 'compressionCodecClassName' from
`parquet.compression`,and the order is
`compression`,`parquet.compression`,`spark.sql.parquet.compression.codec`, just
like what we do in `OrcOptions`.
2.Change `spark.sql.parquet.compression.codec` to support "none".Actually
in `ParquetOptions`,we do support "none" as equivalent to "uncompressed", but
it does not allowed to configured to "none".
## How was this patch tested?
Manual test.
commit 5124f1b560e942c0dc23af31336317a4b995dd8f
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T07:06:26Z
spark.sql.parquet.compression.codec[SPARK-21786][SQL] When acquiring
'compressionCodecClassName' in 'ParquetOptions', `parquet.compression` needs to
be considered.
## What changes were proposed in this pull request?
1.Increased acquiring 'compressionCodecClassName' from
`parquet.compression`,and the order is
`compression`,`parquet.compression`,`spark.sql.parquet.compression.codec`, just
like what we do in `OrcOptions`.
2.Change `spark.sql.parquet.compression.codec` to support "none".Actually
in `ParquetOptions`,we do support "none" as equivalent to "uncompressed", but
it does not allowed to configured to "none".
3.Change `compressionCode` to `compressionCodecClassName`.
## How was this patch tested?
Manual test.
commit 6907a3ef86a2546fae91c22754796490a80effff
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T09:26:33Z
Make comression codec take effect in hive table writing.
commit 67e40d4d7fd3b6a9e4526ce17bf6d4eadb05b2b8
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-25T12:08:11Z
Modify test
commit e2526ca1bb72e54c03d977b8678bd14b28c83585
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-26T05:38:10Z
Separate the pr
commit 8ae86ee11de1e2a4d22bf0e7111478167ce5a8b9
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-26T08:09:11Z
Add test case with the table containing mixed compression codec
commit 94ac716551c48ffd48b1f6590ceeac0a95888490
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-26T08:26:46Z
Revert back
commit 43e041f2b7c350ddafea447a356b4df572ce0b2a
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-26T08:30:12Z
Revert back
commit ee0c5587374ed33e29a763f8764263114a9b57ab
Author: fjh100456 <fu.jinhua6@...>
Date: 2017-12-26T08:33:51Z
Add a new line at the of file
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
