Hi, We have a Apache Beam pipeline running in Google Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our other workers from other pipelines continued to get Kafka messages.
At the moment it stopped we got these messages: I [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException. I [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery I [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) And then the pipeline stopped reading the messages. This is the KafkaIO setup we have: KafkaIO.read[String,String]() .withBootstrapServers(server) .withTopic(topic) .withKeyDeserializer(classOf[StringDeserializer]) .withValueDeserializer(classOf[StringDeserializer]) .updateConsumerProperties(properties) .commitOffsetsInFinalize() .withoutMetadata() Any help will be much appreciated. Best regards, -- Eduardo Soldera Garcia Data Engineer (16) 3509-5555 | www.arquivei.com.br <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> [image: Arquivei.com.br – Inteligência em Notas Fiscais] <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> [image: Google seleciona Arquivei para imersão e mentoria no Vale do Silício] <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> <https://www.facebook.com/arquivei> <https://www.linkedin.com/company/arquivei> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
