On Fri, Oct 19, 2018 at 6:54 AM Eduardo Soldera <
eduardo.sold...@arquivei.com.br> wrote:
> Hi Raghu, just a quick update. We were waiting for Spotify's Scio to
> update to Beam 2.7. We've just deployed the pipeling sucessfully. Just for
> letting you know, I tried to use the workaround code snipped, but Dataflow
> wouldn't recover after a Kafka unavailability.
>
Thanks for the update. The workaround helps only if KafkaClient itself can
recover when try to read again. I guess some of those exceptions are are
not recoverable.
Please let us know how the actual fix works.
Thanks.
Raghu.
>
> Thanks for your help.
>
> Regards
>
> Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <rang...@google.com>
> escreveu:
>
>>
>>
>> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jcgarc...@gmail.com>
>> wrote:
>>
>>> Sorry I hit the send button to fast... The error occurs in the worker.
>>>
>>
>> Np. Just one more comment on it: it is a very important
>> design/correctness decision to for runner to decide how to handle
>> persistent errors in a streaming pipeline. Dataflow keeps failing since
>> there is no solution to restart a pipeline from scratch without losing
>> exactly-once guarantees. It lets user decide if the pipeline needs to be
>> 'upgraded'.
>>
>> Raghu.
>>
>>>
>>> Juan Carlos Garcia <jcgarc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
>>> 20:22:
>>>
>>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>>> yarn retries multiple times until it reachs it max attempt and then gives
>>>> up.
>>>>
>>>> Raghu Angadi <rang...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>>>>
>>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <
>>>>> jcgarc...@gmail.com> wrote:
>>>>>
>>>>>> Don't know if its related, but we have seen our pipeline dying (using
>>>>>> SparkRunner) when there is problem with Kafka (network interruptions),
>>>>>> errors like: org.apache.kafka.common.errors.TimeoutException: Timeout
>>>>>> expired while fetching topic metadata
>>>>>>
>>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>>> *withConsumerFactoryFn.*
>>>>>>
>>>>>
>>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>>> worker? or does it happen while launching the pipeline on the client?
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu, thank you.
>>>>>>>
>>>>>>> I'm not sure though what to pass as an argument:
>>>>>>>
>>>>>>> KafkaIO.read[String,String]()
>>>>>>> .withBootstrapServers(server)
>>>>>>> .withTopic(topic)
>>>>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>>>>> .withConsumerFactoryFn(new
>>>>>>> KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>>> .updateConsumerProperties(properties)
>>>>>>> .commitOffsetsInFinalize()
>>>>>>> .withoutMetadata()
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <rang...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Hi Eduardo,
>>>>>>>>
>>>>>>>> There another work around you can try without having to wait for
>>>>>>>> 2.7.0 release: Use a wrapper to catch exception from
>>>>>>>> KafkaConsumer#poll()
>>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>>
>>>>>>>> Using something like (such a wrapper is used in KafkasIO tests
>>>>>>>> [2]):
>>>>>>>> private static class ConsumerFactoryFn
>>>>>>>> implements SerializableFunction<Map<String,
>>>>>>>> Object>, Consumer<byte[], byte[]>> {
>>>>>>>> @Override
>>>>>>>> public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>>> config) {
>>>>>>>> return new KafkaConsumer(config) {
>>>>>>>> @Override
>>>>>>>> public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>>> // work around for BEAM-5375
>>>>>>>> while (true) {
>>>>>>>> try {
>>>>>>>> return super.poll(timeout);
>>>>>>>> } catch (Exception e) {
>>>>>>>> // LOG & sleep for sec
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>>> [2]:
>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>>
>>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu, we're not sure how long the network was down. According
>>>>>>>>> to the logs no longer than one minute. A 30 second shutdown would
>>>>>>>>> work for
>>>>>>>>> the tests.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>>> down?
>>>>>>>>>>
>>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but
>>>>>>>>>>> if you could send us some guidance would be great.
>>>>>>>>>>>
>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>>> eduardo.sold...@arquivei.com.br> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm
>>>>>>>>>>>> not sure how we'd use the patch. We're using SBT and Spotify's
>>>>>>>>>>>> Scio with
>>>>>>>>>>>> Scala.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I
>>>>>>>>>>>>> can make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that
>>>>>>>>>>>>>>>>>> provides temporary
>>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned
>>>>>>>>>>>>>>>>>> earlier needs to handle unchecked exception. In your case it
>>>>>>>>>>>>>>>>>> is clear that
>>>>>>>>>>>>>>>>>> poll thread exited with a runtime exception. The reader does
>>>>>>>>>>>>>>>>>> not check for
>>>>>>>>>>>>>>>>>> it and continues to wait for poll thread to enqueue
>>>>>>>>>>>>>>>>>> messages. A fix should
>>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The
>>>>>>>>>>>>>>>>>> runners will handle
>>>>>>>>>>>>>>>>>> that appropriately after that. I will file a jira.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>>> kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>>> - Implies the reader is initialized and poll
>>>>>>>>>>>>>>>>>> thread is started.
>>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>>> offset 2
>>>>>>>>>>>>>>>>>> - The reader actually got a message received by
>>>>>>>>>>>>>>>>>> the poll thread from Kafka.
>>>>>>>>>>>>>>>>>> - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>>> while fetching latest offset for partition kafka_topic-0.
>>>>>>>>>>>>>>>>>> will be retried.
>>>>>>>>>>>>>>>>>> - This must have happened around the time when
>>>>>>>>>>>>>>>>>> network was disrupted. This is from. Actual log is
>>>>>>>>>>>>>>>>>> from another periodic
>>>>>>>>>>>>>>>>>> task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The poll thread must have died around the time network
>>>>>>>>>>>>>>>>>> was disrupted.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset.
>>>>>>>>>>>>>>>>>> This log seems to
>>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional
>>>>>>>>>>>>>>>>>> log. I don't
>>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how
>>>>>>>>>>>>>>>>>> KafkaIO does
>>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>>> clientId=consumer-2,
>>>>>>>>>>>>>>>>>> groupId=Reader-0_offset_consumer_1735388161_genericPipe]
>>>>>>>>>>>>>>>>>> Resetting offset
>>>>>>>>>>>>>>>>>> for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying
>>>>>>>>>>>>>>>>> to see in the worker logs. One way to avoid is to set kafka
>>>>>>>>>>>>>>>>> consumer's log
>>>>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something
>>>>>>>>>>>>>>>>> to avoid it
>>>>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running
>>>>>>>>>>>>>>>>>>>> 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at
>>>>>>>>>>>>>>>>>>>> line 594 & 587
>>>>>>>>>>>>>>>>>>>> [1]. Dataflow caches its readers and DirectRunner may
>>>>>>>>>>>>>>>>>>>> not. That can
>>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in
>>>>>>>>>>>>>>>>>>>> KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of
>>>>>>>>>>>>>>>>>>>> connection problems (as
>>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll()
>>>>>>>>>>>>>>>>>>>> throws and we need to
>>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the
>>>>>>>>>>>>>>>>>>>>> pipeline were still
>>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the
>>>>>>>>>>>>>>>>>>>>> current offset stays at 15, the commited offset from the
>>>>>>>>>>>>>>>>>>>>> last processed
>>>>>>>>>>>>>>>>>>>>> message, before the error.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue
>>>>>>>>>>>>>>>>>>>>> in a Dev scenario.
>>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker
>>>>>>>>>>>>>>>>>>>>> network and the same
>>>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the
>>>>>>>>>>>>>>>>>>>>> pipeline was able
>>>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our
>>>>>>>>>>>>>>>>>>>>> production scenario
>>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages.
>>>>>>>>>>>>>>>>>>>>> Unfortunately, we've
>>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the
>>>>>>>>>>>>>>>>>>>>> problematic dev job is
>>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached.
>>>>>>>>>>>>>>>>>>>>> Thank you very much.
>>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be
>>>>>>>>>>>>>>>>>>>>>> one (if a worker
>>>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that
>>>>>>>>>>>>>>>>>>>>>> KafkaClient itself
>>>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error
>>>>>>>>>>>>>>>>>>>>>> (though unlikely).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular
>>>>>>>>>>>>>>>>>>>>>>> error. Do you see any
>>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>>> - login to one of the VMs and get stacktrace of
>>>>>>>>>>>>>>>>>>>>>>> java worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>>> - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop
>>>>>>>>>>>>>>>>>>>>>>>> fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines
>>>>>>>>>>>>>>>>>>>>>>>> continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>>> Error sending fetch request (sessionId=1396189203,
>>>>>>>>>>>>>>>>>>>>>>>> epoch=2431598) to node 3:
>>>>>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>>> Group coordinator 10.0.52.70:9093 (id: 2147483646
>>>>>>>>>>>>>>>>>>>>>>>> rack: null) is unavailable or invalid, will attempt
>>>>>>>>>>>>>>>>>>>>>>>> rediscovery
>>>>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>>>>> Discovered group coordinator 10.0.52.70:9093 (id:
>>>>>>>>>>>>>>>>>>>>>>>> 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup we have:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>>> .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>>> .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>> .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>>> .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>>> .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> JC
>>>>>>
>>>>>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>
