cxzl25 commented on PR #1714:
URL: https://github.com/apache/orc/pull/1714#issuecomment-1999090583
> For Spark community, I'll exclude it officially and add a documentation
later to guide the users to add that dependency from application side.
Currently, the master branch of Spark has used ORC 2.0.0 version and
supports brotli compression encoding. (SPARK-44115)
However, due to some hardcode checks in Spark, brotli cannot be used
normally.
Can I propose a PR that Spark supports ORC brotli ?
https://github.com/cxzl25/spark/commit/0cd5d03116709ec86d9f96e269200f2d05f799ca
```sql
set spark.sql.orc.compression.codec=BROTLI;
```
```
java.lang.IllegalArgumentException: The value of
spark.sql.orc.compression.codec should be one of uncompressed, lz4, lzo,
snappy, zlib, none, zstd, but was brotli
```
By the way, `netty-codec` declares brotli-related dependencies for multiple
OSs.
https://github.com/netty/netty/blob/3cd364107167600e8eb4b0b85553ed895519e2ed/codec/pom.xml#L91-L125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
