pan3793 commented on PR #3114: URL: https://github.com/apache/celeborn/pull/3114#issuecomment-2677924137
I suppose the upgrade should have perf improvements for the Celeborn client because the default value of `celeborn.client.push.buffer.max.size` is `64k` https://github.com/apache/celeborn/blob/f09482108fe91f986b9cfceebd4b050379bf6f64/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala#L4457 and zstd [v1.5.7](https://github.com/facebook/zstd/releases/tag/v1.5.7) release notes say > Enhanced Compression Speed for Small Data Blocks Note: Celeborn Spark shaded client does not ship the zstd-jni and always uses Spark's built-in zstd-jni, if someone wants to test the perf improvement, he/she should upgrade the Spark Spark's built-in zstd-jni first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
