Sorry I hit the send button to fast... The error occurs in the worker.
Juan Carlos Garcia <jcgarc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
20:22:
> Sorry for hijacking the thread, we are running Spark on top of Yarn, yarn
> retries multiple times until it reachs it max attempt and then gives up.
>
> Raghu Angadi <rang...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>
>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jcgarc...@gmail.com>
>> wrote:
>>
>>> Don't know if its related, but we have seen our pipeline dying (using
>>> SparkRunner) when there is problem with Kafka (network interruptions),
>>> errors like: org.apache.kafka.common.errors.TimeoutException: Timeout
>>> expired while fetching topic metadata
>>>
>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>> *withConsumerFactoryFn.*
>>>
>>
>> Wouldn't that be retried by the SparkRunner if it happens on the worker?
>> or does it happen while launching the pipeline on the client?
>>
>>
>>
>>>
>>>
>>>
>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>> eduardo.sold...@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu, thank you.
>>>>
>>>> I'm not sure though what to pass as an argument:
>>>>
>>>> KafkaIO.read[String,String]()
>>>> .withBootstrapServers(server)
>>>> .withTopic(topic)
>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>> .withConsumerFactoryFn(new
>>>> KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>> .updateConsumerProperties(properties)
>>>> .commitOffsetsInFinalize()
>>>> .withoutMetadata()
>>>>
>>>>
>>>> Regards
>>>>
>>>>
>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <rang...@google.com>
>>>> escreveu:
>>>>
>>>>> Hi Eduardo,
>>>>>
>>>>> There another work around you can try without having to wait for 2.7.0
>>>>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>>>>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>
>>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>>> private static class ConsumerFactoryFn
>>>>> implements SerializableFunction<Map<String, Object>,
>>>>> Consumer<byte[], byte[]>> {
>>>>> @Override
>>>>> public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>>>> return new KafkaConsumer(config) {
>>>>> @Override
>>>>> public ConsumerRecords<K, V> poll(long timeout) {
>>>>> // work around for BEAM-5375
>>>>> while (true) {
>>>>> try {
>>>>> return super.poll(timeout);
>>>>> } catch (Exception e) {
>>>>> // LOG & sleep for sec
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> [1]:
>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>> [2]:
>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>
>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, we're not sure how long the network was down. According to
>>>>>> the logs no longer than one minute. A 30 second shutdown would work for
>>>>>> the
>>>>>> tests.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <rang...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>>>>
>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>
>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>>>>> you could send us some guidance would be great.
>>>>>>>>
>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>> eduardo.sold...@arquivei.com.br> escreveu:
>>>>>>>>
>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with
>>>>>>>>> Scala.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>
>>>>>>>>>> Raghu.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>
>>>>>>>>>>> Regards!
>>>>>>>>>>>
>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will fix
>>>>>>>>>>>>> it later this week.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides
>>>>>>>>>>>>>>> temporary
>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear
>>>>>>>>>>>>>>> that poll
>>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not
>>>>>>>>>>>>>>> check for it
>>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A
>>>>>>>>>>>>>>> fix should
>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners
>>>>>>>>>>>>>>> will handle
>>>>>>>>>>>>>>> that appropriately after that. I will file a jira.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>> kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>> - Implies the reader is initialized and poll thread
>>>>>>>>>>>>>>> is started.
>>>>>>>>>>>>>>> - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>> offset 2
>>>>>>>>>>>>>>> - The reader actually got a message received by the
>>>>>>>>>>>>>>> poll thread from Kafka.
>>>>>>>>>>>>>>> - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>>>> fetching latest offset for partition kafka_topic-0. will be
>>>>>>>>>>>>>>> retried.
>>>>>>>>>>>>>>> - This must have happened around the time when
>>>>>>>>>>>>>>> network was disrupted. This is from. Actual log is from
>>>>>>>>>>>>>>> another periodic
>>>>>>>>>>>>>>> task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This
>>>>>>>>>>>>>>> log seems to
>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional
>>>>>>>>>>>>>>> log. I don't
>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how
>>>>>>>>>>>>>>> KafkaIO does
>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>> clientId=consumer-2,
>>>>>>>>>>>>>>> groupId=Reader-0_offset_consumer_1735388161_genericPipe]
>>>>>>>>>>>>>>> Resetting offset
>>>>>>>>>>>>>>> for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka
>>>>>>>>>>>>>> consumer's log
>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to
>>>>>>>>>>>>>> avoid it
>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()'
>>>>>>>>>>>>>>>>> which can
>>>>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594
>>>>>>>>>>>>>>>>> & 587 [1].
>>>>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That
>>>>>>>>>>>>>>>>> can explain
>>>>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that
>>>>>>>>>>>>>>>>> Kafka client
>>>>>>>>>>>>>>>>> library takes care of retrying in case of connection problems
>>>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws
>>>>>>>>>>>>>>>>> and we need to
>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were
>>>>>>>>>>>>>>>>>> still getting
>>>>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last
>>>>>>>>>>>>>>>>>> processed message,
>>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in
>>>>>>>>>>>>>>>>>> a Dev scenario.
>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network
>>>>>>>>>>>>>>>>>> and the same
>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the
>>>>>>>>>>>>>>>>>> pipeline was able
>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner
>>>>>>>>>>>>>>>>>> and did the same test, the same problem from our production
>>>>>>>>>>>>>>>>>> scenario
>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages.
>>>>>>>>>>>>>>>>>> Unfortunately, we've
>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic
>>>>>>>>>>>>>>>>>> dev job is
>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank
>>>>>>>>>>>>>>>>>> you very much.
>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>> rang...@google.com> escreveu:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be
>>>>>>>>>>>>>>>>>>> one (if a worker
>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that
>>>>>>>>>>>>>>>>>>> KafkaClient itself
>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error
>>>>>>>>>>>>>>>>>>> (though unlikely).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rang...@google.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error.
>>>>>>>>>>>>>>>>>>>> Do you see any
>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>> - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>> - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>> eduardo.sold...@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop
>>>>>>>>>>>>>>>>>>>>> fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines
>>>>>>>>>>>>>>>>>>>>> continued to get Kafka
>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>> Error sending fetch request (sessionId=1396189203,
>>>>>>>>>>>>>>>>>>>>> epoch=2431598) to node 3:
>>>>>>>>>>>>>>>>>>>>> org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>> Group coordinator 10.0.52.70:9093 (id: 2147483646 rack:
>>>>>>>>>>>>>>>>>>>>> null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe]
>>>>>>>>>>>>>>>>>>>>> Discovered group coordinator 10.0.52.70:9093 (id:
>>>>>>>>>>>>>>>>>>>>> 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup we have:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>> .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>> .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>> .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>> .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>> .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>> .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>> .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>>
>>> --
>>>
>>> JC
>>>
>>>
