Hi Kamal, That’s what you hope for always; that people first search through documentation. That people search flink code is maybe even beyond that, nice!
In this case I’m no expert. That said, I think the Behind the Scene section on https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene provides some important information: > The split enumerator of Kafka is responsible for discovering new splits > (partitions) under the provided topic partition subscription pattern, and > assigning splits to readers, uniformly distributed across subtasks, in > round-robin style. Note that the split enumerator of Kafka source pushes > splits eagerly to source readers, so it won’t need to handle split requests > from source reader. The code relevant to assigning partitions should be available in https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java. So, in short, Flink seems to organises (assigns) the splits (~ the Flink counterpart of Kafka partitions) over available source readers (in tasks in task executors). As for your question about Cruise control for kafka cluster rebalancing, I think you have things mixed up here. I didn’t know about this LinkedIn project, but apparently it (re)balances partitions over brokers. This somewhat relates to, but is not the same as balancing partitions over consumers. Looping the user mailing list back in as others may chip in or tell me I’m completely wrong ;) As I said, I’m a Flink user, and I’ve worked with Kafka, but I’m far from an expert on the matter. Don’t treat my answer as authoritative in anyway! Best regards, Frens Jan > On 24 Jul 2025, at 13:32, Kamal Mittal <kamal.mit...@ericsson.com> wrote: > > Sure, truly speaking I myself first read the documentation I found and even > searched the flink code for my query but didn’t find anything concrete. > > I am trying to introduce cruise control for kafka cluster rebalancing and due > to that only I asked this query. I even asked about “kafka cluster > rebalancing” for which yet to receive any feedback. > > From: Frens Jan Rumph <frens....@web-iq.com <mailto:frens....@web-iq.com>> > Sent: 24 July 2025 16:41 > To: Kamal Mittal <kamal.mit...@ericsson.com > <mailto:kamal.mit...@ericsson.com>> > Subject: Re: Kafka partition assignment for consumers > > Hi Kamal, > > Thanks for getting back to me! Good to get this conversation starting. I > realise that tone of voice is subjective. There’s no need to be overly > polite, but - at least in my view - some decorum is often helpful. > > Best regards, > Frens Jan > > > > On 24 Jul 2025, at 13:00, Kamal Mittal <kamal.mit...@ericsson.com > <mailto:kamal.mit...@ericsson.com>> wrote: > > Dear Frens, > > Thanks for your message. > > I’m fully aware that this is a community mailing list, and I’ve always > approached it with the intent of constructive discussion and knowledge > sharing. Asking for input on technical questions — even briefly — is not out > of place in such forums. If every message had to meet arbitrary length or > formality standards, participation would likely drop, not improve. > > As for the line you quoted — “Need inputs for below query. Please share > views.” — I don’t see how that’s "unacceptable." It was a concise and polite > request, not a demand. If the brevity offended you, that wasn’t the intent — > but it’s worth recognizing that tone interpretation is subjective, especially > in written communication. > > Lastly, suggesting that someone is treating the list like a "help desk" just > because they asked a question feels unnecessarily harsh. Communities thrive > when people feel welcome to engage, not when they’re policed for wording. > > That said, I’ll certainly continue to be mindful of how my messages are > framed. > > Best, > Kamal > > From: Frens Jan Rumph <frens....@web-iq.com <mailto:frens....@web-iq.com>> > Sent: 24 July 2025 10:25 > To: Kamal Mittal <kamal.mit...@ericsson.com > <mailto:kamal.mit...@ericsson.com>> > Cc: user@flink.apache.org <mailto:user@flink.apache.org> > Subject: Re: Kafka partition assignment for consumers > > Dear Kamal, > > I am not an active member of the Flink community, let alone a moderator or > admin; so I’m probably stepping out of line here. > > And I’m going to be a little bit blunt here, but I don’t think you should > treat this mailing list as a help desk. You need to realise that this is a > community mailing list. Your e-mail ends up in the mailbox of probably > hundreds or maybe even thousands of people. > > I think that you should work on your tone of voice. Something like this is at > least to me unacceptable: > > Need inputs for below query. Please share views. > > Also, I think you should improve your questions themselves. You should > indicate what do you already know, what you did to answer the question > yourself and what documentation was missing. > > In the case of how Flink handles Kafka partition assignment, have you read > the documentation on > https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene? > It even links to some great additional resources. > > Best regards, > Frens Jan > > > > > > > On 24 Jul 2025, at 06:04, Kamal Mittal via user <user@flink.apache.org > <mailto:user@flink.apache.org>> wrote: > > OR may be if you can give inputs that how flink handles consumers scaling? > > From: Kamal Mittal via user <user@flink.apache.org > <mailto:user@flink.apache.org>> > Sent: 24 July 2025 09:16 > To: user@flink.apache.org <mailto:user@flink.apache.org> > Subject: Kafka partition assignment for consumers > > Hello, > > Does Flink assign partitions to consumers for Kafka Source or Kafka itself > does this? > > Rgds, > Kamal
signature.asc
Description: Message signed with OpenPGP
