On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <[email protected]> wrote:
> Don't know if its related, but we have seen our pipeline dying (using > SparkRunner) when there is problem with Kafka (network interruptions), > errors like: org.apache.kafka.common.errors.TimeoutException: Timeout > expired while fetching topic metadata > > Maybe this will fix it as well, thanks Raghu for the hint about > *withConsumerFactoryFn.* > Wouldn't that be retried by the SparkRunner if it happens on the worker? or does it happen while launching the pipeline on the client? > > > > On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera < > [email protected]> wrote: > >> Hi Raghu, thank you. >> >> I'm not sure though what to pass as an argument: >> >> KafkaIO.read[String,String]() >> .withBootstrapServers(server) >> .withTopic(topic) >> .withKeyDeserializer(classOf[StringDeserializer]) >> .withValueDeserializer(classOf[StringDeserializer]) >> .withConsumerFactoryFn(new >> KafkaExecutor.ConsumerFactoryFn(????????????????)) >> .updateConsumerProperties(properties) >> .commitOffsetsInFinalize() >> .withoutMetadata() >> >> >> Regards >> >> >> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <[email protected]> >> escreveu: >> >>> Hi Eduardo, >>> >>> There another work around you can try without having to wait for 2.7.0 >>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and >>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1]. >>> >>> Using something like (such a wrapper is used in KafkasIO tests [2]): >>> private static class ConsumerFactoryFn >>> implements SerializableFunction<Map<String, Object>, >>> Consumer<byte[], byte[]>> { >>> @Override >>> public Consumer<byte[], byte[]> apply(Map<String, Object> config) { >>> return new KafkaConsumer(config) { >>> @Override >>> public ConsumerRecords<K, V> poll(long timeout) { >>> // work around for BEAM-5375 >>> while (true) { >>> try { >>> return super.poll(timeout); >>> } catch (Exception e) { >>> // LOG & sleep for sec >>> } >>> } >>> } >>> } >>> } >>> >>> [1]: >>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417 >>> [2]: >>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261 >>> >>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera < >>> [email protected]> wrote: >>> >>>> Hi Raghu, we're not sure how long the network was down. According to >>>> the logs no longer than one minute. A 30 second shutdown would work for the >>>> tests. >>>> >>>> Regards >>>> >>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <[email protected]> >>>> escreveu: >>>> >>>>> Thanks. I could repro myself as well. How long was the network down? >>>>> >>>>> Trying to get the fix into 2.7 RC2. >>>>> >>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera < >>>>> [email protected]> wrote: >>>>> >>>>>> Just to make myself clear, I'm not sure how to use the patch but if >>>>>> you could send us some guidance would be great. >>>>>> >>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera < >>>>>> [email protected]> escreveu: >>>>>> >>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not >>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with >>>>>>> Scala. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <[email protected]> >>>>>>> escreveu: >>>>>>> >>>>>>>> Is is feasible for you to verify the fix in your dev job? I can >>>>>>>> make a patch against Beam 2.4 branch if you like. >>>>>>>> >>>>>>>> Raghu. >>>>>>>> >>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Raghu, thank you very much for the pull request. >>>>>>>>> We'll wait for the 2.7 Beam release. >>>>>>>>> >>>>>>>>> Regards! >>>>>>>>> >>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi < >>>>>>>>> [email protected]> escreveu: >>>>>>>>> >>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391 >>>>>>>>>> >>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Filed BEAM-5375 >>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will fix >>>>>>>>>>> it later this week. >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following >>>>>>>>>>>>> usual support oncall access protocol that provides temporary >>>>>>>>>>>>> access to >>>>>>>>>>>>> things like logs in GCP): >>>>>>>>>>>>> >>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier >>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear >>>>>>>>>>>>> that poll >>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check >>>>>>>>>>>>> for it >>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix >>>>>>>>>>>>> should >>>>>>>>>>>>> result in an IOException for read from the source. The runners >>>>>>>>>>>>> will handle >>>>>>>>>>>>> that appropriately after that. I will file a jira. >>>>>>>>>>>>> >>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ignore the link.. was pasted here by mistake. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> From the logs (with a comment below each one): >>>>>>>>>>>>> >>>>>>>>>>>>> - 2018-09-12 06:13:07.345 PDT Reader-0: reading from >>>>>>>>>>>>> kafka_topic-0 starting at offset 2 >>>>>>>>>>>>> - Implies the reader is initialized and poll thread is >>>>>>>>>>>>> started. >>>>>>>>>>>>> - 2018-09-12 06:13:07.780 PDT Reader-0: first record >>>>>>>>>>>>> offset 2 >>>>>>>>>>>>> - The reader actually got a message received by the >>>>>>>>>>>>> poll thread from Kafka. >>>>>>>>>>>>> - 2018-09-12 06:16:48.771 PDT Reader-0: exception while >>>>>>>>>>>>> fetching latest offset for partition kafka_topic-0. will be >>>>>>>>>>>>> retried. >>>>>>>>>>>>> - This must have happened around the time when network >>>>>>>>>>>>> was disrupted. This is from. Actual log is from another >>>>>>>>>>>>> periodic task that >>>>>>>>>>>>> fetches latest offsets for partitions. >>>>>>>>>>>>> >>>>>>>>>>>>> The poll thread must have died around the time network was >>>>>>>>>>>>> disrupted. >>>>>>>>>>>>> >>>>>>>>>>>>> The following log comes from kafka client itself and is >>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log >>>>>>>>>>>>> seems to >>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. >>>>>>>>>>>>> I don't >>>>>>>>>>>>> think there is any better to fetch latest offsets than how >>>>>>>>>>>>> KafkaIO does >>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1]. >>>>>>>>>>>>> >>>>>>>>>>>>> - 2018-09-12 06:13:11.786 PDT [Consumer >>>>>>>>>>>>> clientId=consumer-2, >>>>>>>>>>>>> groupId=Reader-0_offset_consumer_1735388161_genericPipe] >>>>>>>>>>>>> Resetting offset >>>>>>>>>>>>> for partition com.arquivei.dataeng.andre-0 to offset 3. >>>>>>>>>>>>> >>>>>>>>>>>>> [1]: >>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to >>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka >>>>>>>>>>>> consumer's log >>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to >>>>>>>>>>>> avoid it >>>>>>>>>>>> without user option. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is >>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi < >>>>>>>>>>>>>> [email protected]> escreveu: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for debugging. >>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace >>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' >>>>>>>>>>>>>>> which can >>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & >>>>>>>>>>>>>>> 587 [1]. >>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can >>>>>>>>>>>>>>> explain >>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that >>>>>>>>>>>>>>> Kafka client >>>>>>>>>>>>>>> library takes care of retrying in case of connection problems >>>>>>>>>>>>>>> (as >>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws >>>>>>>>>>>>>>> and we need to >>>>>>>>>>>>>>> restart the client in KafkaIO. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]: >>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Raghu, thanks for your help. >>>>>>>>>>>>>>>> Just answering your previous question, the following logs >>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were >>>>>>>>>>>>>>>> still getting >>>>>>>>>>>>>>>> the messages, for example: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (...) >>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>>> to offset 10. >>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>>> to offset 15. >>>>>>>>>>>>>>>> ERROR >>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>>> to offset 22. >>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>>> to offset 30. >>>>>>>>>>>>>>>> (...) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current >>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last >>>>>>>>>>>>>>>> processed message, >>>>>>>>>>>>>>>> before the error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a >>>>>>>>>>>>>>>> Dev scenario. >>>>>>>>>>>>>>>> We started the same pipeline using the direct runner, >>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network >>>>>>>>>>>>>>>> and the same >>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the >>>>>>>>>>>>>>>> pipeline was able >>>>>>>>>>>>>>>> to successfully process the subsequent messages. >>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner >>>>>>>>>>>>>>>> and did the same test, the same problem from our production >>>>>>>>>>>>>>>> scenario >>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. >>>>>>>>>>>>>>>> Unfortunately, we've >>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic >>>>>>>>>>>>>>>> dev job is >>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank >>>>>>>>>>>>>>>> you very much. >>>>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi < >>>>>>>>>>>>>>>> [email protected]> escreveu: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread >>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one >>>>>>>>>>>>>>>>> (if a worker >>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that >>>>>>>>>>>>>>>>> KafkaClient itself >>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though >>>>>>>>>>>>>>>>> unlikely). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Eduardo, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying >>>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you >>>>>>>>>>>>>>>>>> see any others >>>>>>>>>>>>>>>>>> afterwards in the log? >>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful >>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>> - login to one of the VMs and get stacktrace of java >>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming) >>>>>>>>>>>>>>>>>> - file a support bug or stackoverflow question with >>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Raghu. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google >>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching >>>>>>>>>>>>>>>>>>> Kafka messages >>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued >>>>>>>>>>>>>>>>>>> to get Kafka >>>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] >>>>>>>>>>>>>>>>>>> Error sending fetch request (sessionId=1396189203, >>>>>>>>>>>>>>>>>>> epoch=2431598) to node 3: >>>>>>>>>>>>>>>>>>> org.apache.kafka.common.errors.DisconnectException. >>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] >>>>>>>>>>>>>>>>>>> Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: >>>>>>>>>>>>>>>>>>> null) is unavailable or invalid, will attempt rediscovery >>>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] >>>>>>>>>>>>>>>>>>> Discovered group coordinator 10.0.52.70:9093 (id: >>>>>>>>>>>>>>>>>>> 2147483646 rack: null) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This is the KafkaIO setup we have: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]() >>>>>>>>>>>>>>>>>>> .withBootstrapServers(server) >>>>>>>>>>>>>>>>>>> .withTopic(topic) >>>>>>>>>>>>>>>>>>> .withKeyDeserializer(classOf[StringDeserializer]) >>>>>>>>>>>>>>>>>>> .withValueDeserializer(classOf[StringDeserializer]) >>>>>>>>>>>>>>>>>>> .updateConsumerProperties(properties) >>>>>>>>>>>>>>>>>>> .commitOffsetsInFinalize() >>>>>>>>>>>>>>>>>>> .withoutMetadata() >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Any help will be much appreciated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e >>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício] >>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria >>>>>>>>>>>>>>>> no Vale do Silício] >>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no >>>>>>>>>>>>>> Vale do Silício] >>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eduardo Soldera Garcia >>>>>>>>> Data Engineer >>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale >>>>>>>>> do Silício] >>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eduardo Soldera Garcia >>>>>>> Data Engineer >>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>>>>>> Silício] >>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>> <https://www.facebook.com/arquivei> >>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Eduardo Soldera Garcia >>>>>> Data Engineer >>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>>>>> Silício] >>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>> <https://www.facebook.com/arquivei> >>>>>> <https://www.linkedin.com/company/arquivei> >>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>> >>>>> >>>> >>>> -- >>>> Eduardo Soldera Garcia >>>> Data Engineer >>>> (16) 3509-5555 | www.arquivei.com.br >>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>>> Silício] >>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>> <https://www.facebook.com/arquivei> >>>> <https://www.linkedin.com/company/arquivei> >>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>> >>> >> >> -- >> Eduardo Soldera Garcia >> Data Engineer >> (16) 3509-5555 | www.arquivei.com.br >> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >> Silício] >> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >> <https://www.facebook.com/arquivei> >> <https://www.linkedin.com/company/arquivei> >> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >> > > > -- > > JC > >
