Hey Lahiru,
As the log message describes, the member is leaving the group because the
consumer
has been closed by your process/application. You need to track down where
in your
application code the Consumer#close method is being called, and why.
There's no technical limit on how many consumers can be in the same
process. Of course
there is a practical limit, eg if you try to start up millions of consumers
then they may struggle
to stay in the group for example due to the millions of background
heartbeat threads competing
for cpu and network resources. But I see no reason why a modern machine
would be unable
to handle just 4 consumers, and anyways the log message you provided makes
it clear that
the consumer is only leaving the group due to being manually closed.
Hope that helps,
Sophie
On Thu, May 6, 2021 at 7:29 AM Lahiru Chandima
wrote:
> Hi,
>
> I am trying to place multiple kafka consumers of a single consumer group in
> the same process. But, I am unable to put more than 3 consumers in the same
> process.The topic which the consumer group tries to read has 10
> partitions.When I try to attach the 4th consumer, it leaves the group after
> the first poll, with below message. No partitions get assigned to the
> consumer.
>
> [ForkJoinPool.commonPool-worker-3] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer
> clientId=consumer-group65-4, groupId=group65] Member
> consumer-group65-4-b148f7ec-b852-46ed-8514-c3af3bd78395 sending LeaveGroup
> request to coordinator my-cluster:9092 (id: 2147483645 rack: null) due to
> the consumer is being closed
>
> What is the reason for this? Is there any way to put more than 3 consumers
> in the same process?
>
> Thank you
> --
> Lahiru Chandima
>