Hi Venkata, Thank you for reporting this concurrency bug and filing the fix.
PR #3776 has been reviewed, merged to both main and branch-0.7, and is included in the new release candidate v0.7.0-rc1. Regarding PR #3745 (scope reducer metadata by partition range), we plan to include it in the next 0.7.x release. Please continue driving the review forward — it's a solid improvement and we'd love to see it land soon. I'll send out the [VOTE] for 0.7.0-rc1 shortly with the updated artifacts. Regards, Zhentao Shuai On 2026/08/05 18:32:50 Venkatakrishnan Sowrirajan wrote: > (Cross posting this in this thread as well) > > Hi Celeborn devs, > > We found a concurrency bug with GetReducerFileGroup in ShuffleClientImpl in > our internal workloads. The blocking RPC runs inside > ConcurrentHashMap.compute, so it holds the per-key bin lock across the > whole round-trip and queues behind every reduce task of the shuffle. At > scale this wedged a stage for over 20 minutes with no fetch activity. I > filed https://github.com/apache/celeborn/pull/3776 to fix it (isolated > change, has a regression test). > > After filing it, I noticed @Chao Sun <[email protected]> already had > https://github.com/apache/celeborn/pull/3687 open for the same issue, and > later superseded it with https://github.com/apache/celeborn/pull/3745, > which scopes the reduceFileGroups fetch to the reader's partition range > instead of shipping the full map to every executor. > > Both are solid improvements: the concurrency fix and the fetch > optimization. I would like to get both into 0.7.0. #3776 is small and ready > for review; happy to rebase it onto the 0.7.0 branch. Would appreciate > thoughts on whether #3745 can make 0.7.0 as well. > > Regards > Venkata krishnan > > > On Wed, Aug 5, 2026 at 8:34 AM kerwin zhang <[email protected]> > wrote: > > > +1 (binding) > > I checked > > - Download links, checksums and PGP signatures are valid. > > - Successfully built the binary from the source on MacOs with Command: > > build/make-distribution.sh --sbt-enabled --release > > > > Regards, > > Kerwin Zhang > > > > Zhentao Shuai <[email protected]> 于2026年8月3日周一 11:06写道： > > > > > > Hello, Celeborn community, > > > > > > This is a call for a vote to release Apache Celeborn > > > 0.7.0-rc0 > > > > > > The git tag to be voted upon: > > > https://github.com/apache/celeborn/releases/tag/v0.7.0-rc0 > > > > > > Source and binary artifacts can be found at: > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.7.0-rc0/ > > > > > > The git commit hash: > > > d6005dbe17d57ba3790435b2dc4976d644772419 > > > > > > The full changelog since the previous release (0.6.3): > > > https://github.com/apache/celeborn/compare/v0.6.3...v0.7.0-rc0 > > > > > > The staging repo: > > > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1135 > > > > > > The fingerprint of the PGP key release artifacts is signed with: > > > 34D5A9C7996B94968C0C2626208CE878499FD25A > > > > > > My public key to verify signatures can be found in: > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS > > > > > > The vote will be open for at least 72 hours or until the necessary > > > number of votes are reached. > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and the reason) > > > > > > Steps to validate the release: > > > https://www.apache.org/info/verification.html > > > > > > - Download links, checksums, and PGP signatures are valid. > > > - Source code distributions have correct names matching the current > > release. > > > - LICENSE and NOTICE files are correct. > > > - All files have license headers if necessary. > > > - No unlicensed compiled archives bundled in the source archive. > > > - The source tarball matches the git tag. > > > - Build from source is successful. > > > > > > Regards, > > > Zaynt Shuai > > >
