ViggoC opened a new issue, #66991: URL: https://github.com/apache/doris/issues/66991
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description ### Background PR #64638 changed cloud colocate tablet placement from modulo hashing to rendezvous (HRW) hashing. This is a good improvement for compute-group scaling because adding or removing one BE only remaps about `1/N` buckets instead of almost all buckets. However, the current [`CloudColocatePlacement.pickBackendId()`](https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudColocatePlacement.java) independently chooses the BE with the highest `murmur3_128(groupId, bucketIdx, beId)` score. It does not consider the number of buckets already assigned to each BE or any capacity bound. As a result, when the bucket-to-BE ratio is small, the placement of an individual colocate group can be noticeably skewed. ### Experiment I ran the current `CloudColocatePlacement` implementation with: - candidate BE IDs: `1..N` - bucket indexes: `0..B-1` - multiple independent group IDs - the same Guava `murmur3_128().newHasher().putLong(groupId).putLong(bucketIdx).putLong(beId)` scoring code as master - metric: `maxPositiveRelativeDeviation = (maxLoad - B/N) / (B/N)` | Buckets / BEs | Average buckets per BE | Mean maximum load | P95 maximum load | P95 maximum positive deviation | |---|---:|---:|---:|---:| | 8 / 3 | 2.67 | 4.06 | 6 | 125.0% | | 16 / 8 | 2.00 | 4.21 | 6 | 200.0% | | 32 / 16 | 2.00 | 4.83 | 6 | 200.0% | | 64 / 16 | 4.00 | 7.89 | 10 | 150.0% | | 128 / 16 | 8.00 | 13.36 | 16 | 100.0% | | 160 / 40 | 4.00 | 8.90 | 11 | 175.0% | | 4096 / 16 | 256.00 | 284.81 | 299 | 16.8% | This is expected for pure Top-1 HRW: for equal-weight BEs, bucket counts are approximately multinomial, and the relative variation becomes large when there are only a few buckets per BE. The existing [unit test](https://github.com/apache/doris/blob/master/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudColocatePlacementTest.java) uses 4096 buckets and focuses on the movement ratio after adding/removing a BE. It does not cover the maximum placement load for small bucket-to-BE ratios. This issue focuses only on **bucket/tablet-count placement skew**. ### Solution ### Possible direction Could we consider a deterministic bounded-load HRW policy when building the placement cache for a `(colocate group, compute group)`? One possible policy is: 1. Compute the complete HRW candidate ranking for every bucket. 2. Keep the first HRW candidate if it is below the configured/calculated capacity bound. 3. Otherwise, select the next HRW-ranked BE that is below the bound. 4. Use a deterministic bucket order and tie-breaking rule so all FEs produce the same placement. The integer bound could be derived from the expected topology and a failure budget: ``` bound = ceil(bucketCount / (targetBeCount - failureBudget)) ``` For example, with 128 buckets, 16 target BEs, and a one-BE failure budget: ``` bound = ceil(128 / 15) = 9 ``` The maximum bucket-count deviation in the normal 16-BE topology would then be bounded at `(9 - 8) / 8 = 12.5%`, while the remaining 15 BEs still have enough total capacity after one BE becomes unavailable. There is a trade-off: rebuilding bounded placement from scratch can move more buckets than pure HRW because capacity decisions may cascade. A sticky variant could preserve valid existing assignments and only reassign buckets on removed or overloaded BEs. If that behavior must survive FE restarts, persistent placement or persistent sparse overrides may be needed because the current placement cache is local-only. ### Short-term mitigation If the placement algorithm is not changed for now, we should recommend using a sufficiently large `bucketCount / beCount` ratio for cloud colocate groups. Increasing the bucket count gives HRW more independent placement choices and reduces the relative tablet-count skew. This is only a statistical mitigation and does not provide a strict upper bound. In the experiment above, ratios between 2 and 8 still show large P95 maximum positive deviations, while `4096 / 16 = 256` reduces the P95 deviation to 16.8%. The appropriate ratio should be selected according to the acceptable maximum deviation, but users should avoid creating colocate groups with only a few buckets per BE. ### Questions 1. Should cloud colocate placement provide an optional or default maximum bucket-count bound? 2. How much additional remapping is acceptable in exchange for a strict placement bound? Related: #64638 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
