Anyone familiar with kafka connector can help review this FLIP? I am looking forward for your reply.
Best Hongshun On Thu, Jul 24, 2025 at 8:13 PM Leonard Xu <[email protected]> wrote: > Thanks Hongshun for driving this work. > > > We also suffering the issue in production Kafka restoration usage, current > design is a nice tradeoff and has considered the new Source implementation > details, +1 from my side. > > > Best, > Leonard > > > > > 2025 7月 19 18:59，Hongshun Wang <[email protected]> 写道： > > > > Hi devs, > > > > I'd like to initiate a discussion about [FLIP-537: Enumerator with Global > > Split Assignment Distribution for Balanced Split Assignment] [1], which > > addresses critical limitations in our current Kafka connector split > > distribution mechanism. > > > > As documented in [FLINK-31762] [2], several scenarios currently lead to > > uneven Kafka split distribution, causing reader delays and performance > > bottlenecks. The core issue stems from the enumerator's lack of > visibility > > into post-assignment split distribution. > > > > This flip does two things: > > 1. ReaderRegistrationEvent Enhancement: SourceOperator should send > > ReaderRegistrationEvent with assigned splits metadata after startup to > > ensure state consistency. > > 2. Implementation in the Kafka connector to resolve imbalanced splits and > > state awareness during recovery (the enumerator will always choose the > > least assigned subtask，and reason aslo as follows) > > > > Any additional questions regarding this FLIP? Looking forward to hearing > > from you. > > > > Best > > Hongshun > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-537%3A+Enumerator+with+Global+Split+Assignment+Distribution+for+Balanced+Split+assignment > > [2] https://issues.apache.org/jira/browse/FLINK-31762 > >
