Hi Celeborn community,
I'd like to kick off the 0.7.0 release now that we're at the end of July.
It has been about 10 months since 0.6.0, and master has accumulated a
substantial set of new features, improvements, and fixes since 0.6.3 alone.
Highlights include:
● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0 and
Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and Hadoop 2
client support.
● C++ native client (CIP-14). Full push/merge data path, revive, LZ4/ZSTD
compression, retry logic, heartbeat decoding, native transport, and
C++-write/Java-read hybrid integration tests — a major push toward feature
parity with the Java client.
● Encryption at Rest (CIP-22). Spark-side implementation landed.
● Shuffle integrity. End-to-end integrity check for Flink, iterator
fully-consumed validation, and extended E2E checked zone.
● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider
support, DFS replication factor configuration, MapPartitionData on DFS, and
multiple robustness fixes.
● Observability. Flush latency per storage tier, eviction counts, read buffer
availability, metadata operation metrics, and more — all reflected in the
Grafana dashboards.
● Performance. Zero-copy sendfile in Netty native transports, parallelized open
stream and commit paths, and reduced lock contention on hot paths.
● Durability & reliability. fsync on commit, automatic RocksDB restoration,
disk-full rejection, Ratis stepdown and outbox retry, and numerous
race-condition and resource-leak fixes.
● Security. RPC authorization, Java deserialization filter, and configurable
HTTP auth bypass paths.
● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
Looking ahead: 1.0.0
With 0.7.0 we are essentially completing the feature surface for a 1.0. I'd
like to propose that the next major release after 0.7.0 will be 1.0.0 — marking
Celeborn's technical maturity and production-grade stability, reflecting the
collective work since graduation: Spark stage rerun, MapReduce and Flink
support, memory and remote storage, Flink hybrid shuffle, HARD_SPLIT in
PushMergedData, skew partition optimization, the C++ client, encryption at
rest, and broad real-world deployment feedback.
I'm volunteering to serve as the release manager for 0.7.0. If there are no
objections, I'll start preparing the release branch and the first RC after this
thread concludes (72 hours, per convention).
Regards,
Zhentao Shuai
