I would suggest you will check you bootstrap definition and server.properties, somehow it looks for http://ip:9092 , kafka is not using http protocol, seems something not configured correctly
בתאריך יום ג׳, 13 ביולי 2021, 21:46, מאת Rahul Patwari < rahulpatwari8...@gmail.com>: > Hi, > > We are facing an issue in our application where Kafka Consumer Retries are > failing whereas a restart of the application is making the Kafka Consumers > work as expected again. > > Kafka Server version is 2.5.0 - confluent 5.5.0 > Kafka Client Version is 2.4.1 - > > {"component":"org.apache.kafka.common.utils.AppInfoParser$AppInfo","message":"Kafka > version: 2.4.1","method":"<init>"} > > Occasionally(every 24 hours), we have observed that the Kafka consumption > rate went down(NOT 0) and the following logs were observed: Generally, the > consumption rate across all consumers is 1k records/sec. When this issue > occurred, the consumption rate dropped to < 100 records/sec > > org.apache.kafka.common.errors.DisconnectException: null > > > {"time":"2021-07-07T22:13:37,385","severity":"INFO","component":"org.apache.kafka.clients.FetchSessionHandler","message":"[Consumer > clientId=consumer-XXXXXX-3, groupId=XXXXXX] Error sending fetch request > (sessionId=405798138, epoch=5808) to node 8: {}.","method":"handleError"} > > org.apache.kafka.common.errors.TimeoutException: Failed > > > {"time":"2021-07-07T22:26:41,379","severity":"INFO","component":"org.apache.kafka.clients.consumer.internals.AbstractCoordinator","message":"[Consumer > clientId=consumer-XXXXXX-3, groupId=XXXXXX] Group coordinator x.x.x.x:9092 > <http://100.98.40.16:9092/> (id: 2147483623 rack: null) is unavailable or > invalid, will attempt rediscovery","method":"markCoordinatorUnknown"} > > > {"time":"2021-07-07T22:27:10,465","severity":"INFO","component":"org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler","message":"[Consumer > clientId=consumer-XXXXXX-3, groupId=XXXXXX] Discovered group coordinator x > .x.x.x:9092 <http://100.98.40.16:9092/> (id: 2147483623 rack: > null)","method":"onSuccess"} > > The consumers retried for more than an hour but the above logs are observed > again. > The consumers started pulling data after a manual restart of the > application. > > No WARN or ERROR logs were observed in Kafka or Zookeeper during this > period. > > Our observation from this incident is that Kafka Consumer retries could not > resolve the issue but a manual restart of the application does. > > Has anyone faced this issue before? Any pointers are appreciated. > > Regards, > Rahul >