Hi Raghu, just a quick update. We were waiting for Spotify's Scio to update
to Beam 2.7. We've just deployed the pipeling sucessfully. Just for letting
you know, I tried to use the workaround code snipped, but Dataflow wouldn't
recover after a Kafka unavailability.
Thanks for your help.
Regards
Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <rang...@google.com>
escreveu:
>
>
> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jcgarc...@gmail.com>
> wrote:
>
>> Sorry I hit the send button to fast... The error occurs in the worker.
>>
>
> Np. Just one more comment on it: it is a very important design/correctness
> decision to for runner to decide how to handle persistent errors in a
> streaming pipeline. Dataflow keeps failing since there is no solution to
> restart a pipeline from scratch without losing exactly-once guarantees. It
> lets user decide if the pipeline needs to be 'upgraded'.
>
> Raghu.
>
>>
>> Juan Carlos Garcia <jcgarc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
>> 20:22:
>>
>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>> yarn retries multiple times until it reachs it max attempt and then gives
>>> up.
>>>
>>> Raghu Angadi <rang...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>>>
>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jcgarc...@gmail.com>
>>>> wrote:
>>>>
>>>>> Don't know if its related, but we have seen our pipeline dying (using
>>>>> SparkRunner) when there is problem with Kafka (network interruptions),
>>>>> errors like: org.apache.kafka.common.errors.TimeoutException: Timeout
>>>>> expired while fetching topic metadata
>>>>>
>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>> *withConsumerFactoryFn.*
>>>>>
>>>>
>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>> worker? or does it happen while launching the pipeline on the client?
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, thank you.
>>>>>>
>>>>>> I'm not sure though what to pass as an argument:
>>>>>>
>>>>>> KafkaIO.read[String,String]()
>>>>>> .withBootstrapServers(server)
>>>>>> .withTopic(topic)
>>>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>>>> .withConsumerFactoryFn(new
>>>>>> KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>> .updateConsumerProperties(properties)
>>>>>> .commitOffsetsInFinalize()
>>>>>> .withoutMetadata()
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <rang...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Hi Eduardo,
>>>>>>>
>>>>>>> There another work around you can try without having to wait for
>>>>>>> 2.7.0 release: Use a wrapper to catch exception from
>>>>>>> KafkaConsumer#poll()
>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>
>>>>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>>>>> private static class ConsumerFactoryFn
>>>>>>> implements SerializableFunction<Map<String, Object>,
>>>>>>> Consumer<byte[], byte[]>> {
>>>>>>> @Override
>>>>>>> public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>> config) {
>>>>>>> return new KafkaConsumer(config) {
>>>>>>> @Override
>>>>>>> public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>> // work around for BEAM-5375
>>>>>>> while (true) {
>>>>>>> try {
>>>>>>> return super.poll(timeout);
>>>>>>> } catch (Exception e) {
>>>>>>> // LOG & sleep for sec
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>> [2]:
>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>
>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, we're not sure how long the network was down. According
>>>>>>>> to the logs no longer than one minute. A 30 second shutdown would work
>>>>>>>> for
>>>>>>>> the tests.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>> down?
>>>>>>>>>
>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but
>>>>>>>>>> if you could send us some guidance would be great.
>>>>>>>>>>
>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>> eduardo.sold...@arquivei.com.br> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio
>>>>>>>>>>> with Scala.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>
>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides
>>>>>>>>>>>>>>>>> temporary
>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear
>>>>>>>>>>>>>>>>> that poll
>>>>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not
>>>>>>>>>>>>>>>>> check for it
>>>>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A
>>>>>>>>>>>>>>>>> fix should
>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The
>>>>>>>>>>>>>>>>> runners will handle
>>>>>>>>>>>>>>>>> that appropriately after that. I will file a jira.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>> kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>> - Implies the reader is initialized and poll thread
>>>>>>>>>>>>>>>>> is started.
>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>> offset 2
>>>>>>>>>>>>>>>>> - The reader actually got a message received by the
>>>>>>>>>>>>>>>>> poll thread from Kafka.
>>>>>>>>>>>>>>>>> - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>> while fetching latest offset for partition kafka_topic-0.
>>>>>>>>>>>>>>>>> will be retried.
>>>>>>>>>>>>>>>>> - This must have happened around the time when
>>>>>>>>>>>>>>>>> network was disrupted. This is from. Actual log is from
>>>>>>>>>>>>>>>>> another periodic
>>>>>>>>>>>>>>>>> task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This
>>>>>>>>>>>>>>>>> log seems to
>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional
>>>>>>>>>>>>>>>>> log. I don't
>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how
>>>>>>>>>>>>>>>>> KafkaIO does
>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>> clientId=consumer-2,
>>>>>>>>>>>>>>>>> groupId=Reader-0_offset_consumer_1735388161_genericPipe]
>>>>>>>>>>>>>>>>> Resetting offset
>>>>>>>>>>>>>>>>> for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying
>>>>>>>>>>>>>>>> to see in the worker logs. One way to avoid is to set kafka
>>>>>>>>>>>>>>>> consumer's log
>>>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something
>>>>>>>>>>>>>>>> to avoid it
>>>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running
>>>>>>>>>>>>>>>>>>> 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at
>>>>>>>>>>>>>>>>>>> line 594 & 587
>>>>>>>>>>>>>>>>>>> [1]. Dataflow caches its readers and DirectRunner may not.
>>>>>>>>>>>>>>>>>>> That can
>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in
>>>>>>>>>>>>>>>>>>> KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection
>>>>>>>>>>>>>>>>>>> problems (as
>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll()
>>>>>>>>>>>>>>>>>>> throws and we need to
>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline
>>>>>>>>>>>>>>>>>>>> were still
>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last
>>>>>>>>>>>>>>>>>>>> processed message,
>>>>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue
>>>>>>>>>>>>>>>>>>>> in a Dev scenario.
>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker
>>>>>>>>>>>>>>>>>>>> network and the same
>>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the
>>>>>>>>>>>>>>>>>>>> pipeline was able
>>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our
>>>>>>>>>>>>>>>>>>>> production scenario
>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages.
>>>>>>>>>>>>>>>>>>>> Unfortunately, we've
>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the
>>>>>>>>>>>>>>>>>>>> problematic dev job is
>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached.
>>>>>>>>>>>>>>>>>>>> Thank you very much.
>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be
>>>>>>>>>>>>>>>>>>>>> one (if a worker
>>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that
>>>>>>>>>>>>>>>>>>>>> KafkaClient itself
>>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error
>>>>>>>>>>>>>>>>>>>>> (though unlikely).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular
>>>>>>>>>>>>>>>>>>>>>> error. Do you see any
>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>> - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>> - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop
>>>>>>>>>>>>>>>>>>>>>>> fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines
>>>>>>>>>>>>>>>>>>>>>>> continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>> Error sending fetch request (sessionId=1396189203,
>>>>>>>>>>>>>>>>>>>>>>> epoch=2431598) to node 3:
>>>>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>> Group coordinator 10.0.52.70:9093 (id: 2147483646 rack:
>>>>>>>>>>>>>>>>>>>>>>> null) is unavailable or invalid, will attempt
>>>>>>>>>>>>>>>>>>>>>>> rediscovery
>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>> Discovered group coordinator 10.0.52.70:9093 (id:
>>>>>>>>>>>>>>>>>>>>>>> 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup we have:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>> .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>> .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>> .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>> .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>> .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> JC
>>>>>
>>>>>
--
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>
