Thanks Hongshun for driving this work.

We also suffering the issue in production Kafka restoration usage, current 
design is a nice tradeoff and has considered the new Source implementation 
details, +1 from my side.


Best,
Leonard



> 2025 7月 19 18:59,Hongshun Wang <loserwang1...@gmail.com> 写道:
> 
> Hi devs,
> 
> I'd like to initiate a discussion about [FLIP-537: Enumerator with Global
> Split Assignment Distribution for Balanced Split Assignment] [1], which
> addresses critical limitations in our current Kafka connector split
> distribution mechanism.
> 
> As documented in [FLINK-31762] [2], several scenarios currently lead to
> uneven Kafka split distribution, causing reader delays and performance
> bottlenecks. The core issue stems from the enumerator's lack of visibility
> into post-assignment split distribution.
> 
> This flip does two things:
> 1. ReaderRegistrationEvent Enhancement: SourceOperator should send
> ReaderRegistrationEvent with assigned splits metadata after startup to
> ensure state consistency.
> 2. Implementation in the Kafka connector to resolve imbalanced splits and
> state awareness during recovery (the enumerator will always choose the
> least assigned subtask,and reason aslo as follows)
> 
> Any additional questions regarding this FLIP? Looking forward to hearing
> from you.
> 
> Best
> Hongshun
> 
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-537%3A+Enumerator+with+Global+Split+Assignment+Distribution+for+Balanced+Split+assignment
> [2] https://issues.apache.org/jira/browse/FLINK-31762

Reply via email to