dongjoon-hyun commented on pull request #32826: URL: https://github.com/apache/spark/pull/32826#issuecomment-860062911
@dchristle . I'm a big supporter of ZStandard and have no doubt that we need to upgrade ZSTD-JNI in the future. Your PR will be a part of Apache Spark definitely. - https://databricks.com/session_na21/the-rise-of-zstandard-apache-spark-parquet-orc-avro Here, I'm saying that what we need for Apache Spark. What we need is the actual verification by testing, not a hunch. Both of us don't want to break Apache Spark 3.2.0, do we? As you see SPARK-34651, I did it before for ZSTD-JNI 1.4.9-1 and Apache Spark 3.2.0. For your PR, we can proceed in this way. First of all, let's make it sure that ZSTD-JNI 1.5 passes all UTs of Parquet/Kafka/Avro at least. Second, let's merge your Apache Spark PR first temporarily for the wider Apache Spark community testing. If something broken is found during Apache Spark 3.2.0 QA period, we can revert it during that period. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
