[GitHub] [spark] dchristle commented on pull request #32826: [SPARK-35670][BUILD] Upgrade ZSTD-JNI to 1.5.0-1

[GitBox]

dchristle commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-859115935


   > Thank you for your efforts. BTW, @dchristle . Please note that your ORC PR 
is not about ZSTD-JNI. It's native ZSTD library only. I commented on your ORC 
PR about the difference.
   > 
   > For the following, I saw Kafka failures.
   > 
   > > They appear to pass their respective CIs.
   > 
   > <img alt="Screen Shot 2021-06-08 at 11 16 21 PM" width="852" 
src="https://user-images.githubusercontent.com/9700541/121303167-9afc7380-c8af-11eb-9e2e-7500a3467325.png";>
   > 
   > No worry~ For Apache Avro, they have a dependency bot. I guess they will 
catch up soon. Let's wait and see their activity.
   > 
   > > I have less familiarity with Avro's build chains/codebase, so I did not 
attempt to test it yet.
   > 
   > In addition, all libraries should be synced inside Apache Spark because 
Apache Spark is using everything.
   
   Yes, for ORC it's the native C library and not Java. I have a tangential 
question for you: Does it make sense to use `aircompressor` for ZSTD in ORC, 
rather than the `zstd-jni`? It does not seem to keep up with the latest `zstd`, 
and the implementation seems to lack support for many of the strategies 
employed at different compression levels, if I understand the code here 
https://github.com/airlift/aircompressor/blob/495bae80ac7487d2efa1bba437d04e8a2a42bb7b/src/main/java/io/airlift/compress/zstd/CompressionParameters.java#L143
 correctly.
   
   The reason I ask is because it is conceivable that `zstd` in the future 
makes an incompatible change that propagates to `zstd-jni` but not 
`aircompressor`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org