Hongshun Wang created FLINK-41564:
-------------------------------------
Summary: Improve bucket assignment balance across partitions in
Fluss source
Key: FLINK-41564
URL: https://issues.apache.org/jira/browse/FLINK-41564
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Affects Versions: cdc-3.7.0
Reporter: Hongshun Wang
Fix For: cdc-3.8.0
{{FlinkSourceEnumerator#getSplitOwner()}} currently calculates a separate
starting subtask for each partition using its partition ID:
{code:java}
startIndex = ((partitionId.hashCode() * 31) & 0x7FFFFFFF) % parallelism;
owner = (startIndex + bucketId) % parallelism;
{code}
This does not guarantee balanced bucket assignment across partitions. Partition
IDs are globally allocated and may be sparse within a table. Multiple
partitions can therefore have the same starting subtask, leaving some readers
overloaded while others are idle.
For example, with source parallelism {{{}4{}}}, one bucket per partition, and
partition IDs {{{}0, 4, 8, 12{}}}, the current bucket distribution is:
{code:java}
Current: [4, 0, 0, 0]
Expected: [1, 1, 1, 1]
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)