[GitHub] [spark] dchristle commented on pull request #32826: [SPARK-35670][BUILD] Upgrade ZSTD-JNI to 1.5.0-1

[GitBox]

dchristle commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-858146238


   > Thank you for your efforts. BTW, @dchristle . Please note that your ORC PR 
is not about ZSTD-JNI. It's native ZSTD library only. I commented on your ORC 
PR about the difference.
   > 
   > 
   > 
   > For the following, I saw Kafka failures.
   > 
   > > They appear to pass their respective CIs.
   > 
   > <img width="852" alt="Screen Shot 2021-06-08 at 11 16 21 PM" 
src="https://user-images.githubusercontent.com/9700541/121303167-9afc7380-c8af-11eb-9e2e-7500a3467325.png";>
   > 
   > 
   > 
   > No worry~ For Apache Avro, they have a dependency bot. I guess they will 
catch up soon. Let's wait and see their activity.
   > 
   > > I have less familiarity with Avro's build chains/codebase, so I did not 
attempt to test it yet.
   > 
   > 
   > 
   > In addition, all libraries should be synced inside Apache Spark because 
Apache Spark is using everything.
   
   Thank you for your message. The Kafka PR failures seem to not be related to 
the Zstd change -- the tests appear to be bugged/flaky, as many other recent 
PRs are also failing. I looked over the Zstd release notes but did not catch 
any obvious big changes that could trigger an incompatibility. However, my plan 
is to wait till the Kafka PR can pass the CI, and shepherd the change through. 
   
   Regarding Spark: Is it necessary to have all dependencies upgraded to Zstd 
1.5.x before merging if the Spark CI/dependency tests appear to pass? For 
instance, the move to 1.5.0-1 is a scheduled for Kafka 3.0 (unless there is a 
back-port), but I imagine that release will be some time from now. 
   
   Regarding this PR: Do we have a good understanding of why the benchmark 
tests fail? I cannot tell if it is actually related to this code change. 
   
   Thank you for your guidance with this process. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org