Thanks for the details. So query is more around "rebalancing" which is done by kafka itself in case of consumer scaling and by cruise control incase of broker scaling.
During consumer scaling, partition redistribution is handled by flink without causing any issue with partition splits and consumer offsets? Also as redistribution might take some time depending upon partition size so flink kafka consumer may need to hold for read? In this case kafka is given responsibility to assign partitions as application only gives topics configuration to flink kafka source. During broker scaling, partition (replica) movement happens and, in that case, flink kafka source and sink will be able to handle this replica movement? Let me know please if misunderstood something. ________________________________ Hi Kamal, That’s what you hope for always; that people first search through documentation. That people search flink code is maybe even beyond that, nice! In this case I’m no expert. That said, I think the Behind the Scene section on https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene provides some important information: The split enumerator of Kafka is responsible for discovering new splits (partitions) under the provided topic partition subscription pattern, and assigning splits to readers, uniformly distributed across subtasks, in round-robin style. Note that the split enumerator of Kafka source pushes splits eagerly to source readers, so it won’t need to handle split requests from source reader. The code relevant to assigning partitions should be available in https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java. So, in short, Flink seems to organises (assigns) the splits (~ the Flink counterpart of Kafka partitions) over available source readers (in tasks in task executors). As for your question about Cruise control for kafka cluster rebalancing, I think you have things mixed up here. I didn’t know about this LinkedIn project, but apparently it (re)balances partitions over brokers. This somewhat relates to, but is not the same as balancing partitions over consumers. Looping the user mailing list back in as others may chip in or tell me I’m completely wrong ;) As I said, I’m a Flink user, and I’ve worked with Kafka, but I’m far from an expert on the matter. Don’t treat my answer as authoritative in anyway! Best regards, Frens Jan On 24 Jul 2025, at 13:32, Kamal Mittal <kamal.mit...@ericsson.com> wrote: Sure, truly speaking I myself first read the documentation I found and even searched the flink code for my query but didn’t find anything concrete. I am trying to introduce cruise control for kafka cluster rebalancing and due to that only I asked this query. I even asked about “kafka cluster rebalancing” for which yet to receive any feedback. From: Frens Jan Rumph <frens....@web-iq.com<mailto:frens....@web-iq.com>> Sent: 24 July 2025 16:41 To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Subject: Re: Kafka partition assignment for consumers Hi Kamal, Thanks for getting back to me! Good to get this conversation starting. I realise that tone of voice is subjective. There’s no need to be overly polite, but - at least in my view - some decorum is often helpful. Best regards, Frens Jan On 24 Jul 2025, at 13:00, Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> wrote: Dear Frens, Thanks for your message. I’m fully aware that this is a community mailing list, and I’ve always approached it with the intent of constructive discussion and knowledge sharing. Asking for input on technical questions — even briefly — is not out of place in such forums. If every message had to meet arbitrary length or formality standards, participation would likely drop, not improve. As for the line you quoted — “Need inputs for below query. Please share views.” — I don’t see how that’s "unacceptable." It was a concise and polite request, not a demand. If the brevity offended you, that wasn’t the intent — but it’s worth recognizing that tone interpretation is subjective, especially in written communication. Lastly, suggesting that someone is treating the list like a "help desk" just because they asked a question feels unnecessarily harsh. Communities thrive when people feel welcome to engage, not when they’re policed for wording. That said, I’ll certainly continue to be mindful of how my messages are framed. Best, Kamal From: Frens Jan Rumph <frens....@web-iq.com<mailto:frens....@web-iq.com>> Sent: 24 July 2025 10:25 To: Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Kafka partition assignment for consumers Dear Kamal, I am not an active member of the Flink community, let alone a moderator or admin; so I’m probably stepping out of line here. And I’m going to be a little bit blunt here, but I don’t think you should treat this mailing list as a help desk. You need to realise that this is a community mailing list. Your e-mail ends up in the mailbox of probably hundreds or maybe even thousands of people. I think that you should work on your tone of voice. Something like this is at least to me unacceptable: > Need inputs for below query. Please share views. Also, I think you should improve your questions themselves. You should indicate what do you already know, what you did to answer the question yourself and what documentation was missing. In the case of how Flink handles Kafka partition assignment, have you read the documentation on https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene? It even links to some great additional resources. Best regards, Frens Jan On 24 Jul 2025, at 06:04, Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> wrote: OR may be if you can give inputs that how flink handles consumers scaling? From: Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> Sent: 24 July 2025 09:16 To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Kafka partition assignment for consumers Hello, Does Flink assign partitions to consumers for Kafka Source or Kafka itself does this? Rgds, Kamal
