You would pass a SerializableFunction (with whatever args required to create the object of this functor, if any). E.g. ConsumerFactoryFn() defined in my pseudo code below does not need to any args. So it would be : ------------ KafkaIO.read() ... .withConsumerFactoryFn(new ConsumerFactoryFn()). -----------
On Wed, Sep 19, 2018 at 6:29 AM Eduardo Soldera < [email protected]> wrote: > Hi Raghu, thank you. > > I'm not sure though what to pass as an argument: > > KafkaIO.read[String,String]() > .withBootstrapServers(server) > .withTopic(topic) > .withKeyDeserializer(classOf[StringDeserializer]) > .withValueDeserializer(classOf[StringDeserializer]) > .withConsumerFactoryFn(new > KafkaExecutor.ConsumerFactoryFn(????????????????)) > .updateConsumerProperties(properties) > .commitOffsetsInFinalize() > .withoutMetadata() > > > Regards > > > Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <[email protected]> > escreveu: > >> Hi Eduardo, >> >> There another work around you can try without having to wait for 2.7.0 >> release: Use a wrapper to catch exception from KafkaConsumer#poll() and >> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1]. >> >> Using something like (such a wrapper is used in KafkasIO tests [2]): >> private static class ConsumerFactoryFn >> implements SerializableFunction<Map<String, Object>, >> Consumer<byte[], byte[]>> { >> @Override >> public Consumer<byte[], byte[]> apply(Map<String, Object> config) { >> return new KafkaConsumer(config) { >> @Override >> public ConsumerRecords<K, V> poll(long timeout) { >> // work around for BEAM-5375 >> while (true) { >> try { >> return super.poll(timeout); >> } catch (Exception e) { >> // LOG & sleep for sec >> } >> } >> } >> } >> } >> >> [1]: >> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417 >> [2]: >> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261 >> >> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera < >> [email protected]> wrote: >> >>> Hi Raghu, we're not sure how long the network was down. According to the >>> logs no longer than one minute. A 30 second shutdown would work for the >>> tests. >>> >>> Regards >>> >>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <[email protected]> >>> escreveu: >>> >>>> Thanks. I could repro myself as well. How long was the network down? >>>> >>>> Trying to get the fix into 2.7 RC2. >>>> >>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera < >>>> [email protected]> wrote: >>>> >>>>> Just to make myself clear, I'm not sure how to use the patch but if >>>>> you could send us some guidance would be great. >>>>> >>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera < >>>>> [email protected]> escreveu: >>>>> >>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure >>>>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <[email protected]> >>>>>> escreveu: >>>>>> >>>>>>> Is is feasible for you to verify the fix in your dev job? I can make >>>>>>> a patch against Beam 2.4 branch if you like. >>>>>>> >>>>>>> Raghu. >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Raghu, thank you very much for the pull request. >>>>>>>> We'll wait for the 2.7 Beam release. >>>>>>>> >>>>>>>> Regards! >>>>>>>> >>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi < >>>>>>>> [email protected]> escreveu: >>>>>>>> >>>>>>>>> Fix: https://github.com/apache/beam/pull/6391 >>>>>>>>> >>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. >>>>>>>>>> I will fix it later this week. >>>>>>>>>> >>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following >>>>>>>>>>>> usual support oncall access protocol that provides temporary >>>>>>>>>>>> access to >>>>>>>>>>>> things like logs in GCP): >>>>>>>>>>>> >>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier >>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that >>>>>>>>>>>> poll >>>>>>>>>>>> thread exited with a runtime exception. The reader does not check >>>>>>>>>>>> for it >>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix >>>>>>>>>>>> should >>>>>>>>>>>> result in an IOException for read from the source. The runners >>>>>>>>>>>> will handle >>>>>>>>>>>> that appropriately after that. I will file a jira. >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ignore the link.. was pasted here by mistake. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From the logs (with a comment below each one): >>>>>>>>>>>> >>>>>>>>>>>> - 2018-09-12 06:13:07.345 PDT Reader-0: reading from >>>>>>>>>>>> kafka_topic-0 starting at offset 2 >>>>>>>>>>>> - Implies the reader is initialized and poll thread is >>>>>>>>>>>> started. >>>>>>>>>>>> - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset >>>>>>>>>>>> 2 >>>>>>>>>>>> - The reader actually got a message received by the poll >>>>>>>>>>>> thread from Kafka. >>>>>>>>>>>> - 2018-09-12 06:16:48.771 PDT Reader-0: exception while >>>>>>>>>>>> fetching latest offset for partition kafka_topic-0. will be >>>>>>>>>>>> retried. >>>>>>>>>>>> - This must have happened around the time when network >>>>>>>>>>>> was disrupted. This is from. Actual log is from another >>>>>>>>>>>> periodic task that >>>>>>>>>>>> fetches latest offsets for partitions. >>>>>>>>>>>> >>>>>>>>>>>> The poll thread must have died around the time network was >>>>>>>>>>>> disrupted. >>>>>>>>>>>> >>>>>>>>>>>> The following log comes from kafka client itself and is printed >>>>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to >>>>>>>>>>>> be added >>>>>>>>>>>> in recent versions. It is probably an unintentional log. I don't >>>>>>>>>>>> think >>>>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does >>>>>>>>>>>> now. This >>>>>>>>>>>> is logged inside consumer.position() called at [1]. >>>>>>>>>>>> >>>>>>>>>>>> - 2018-09-12 06:13:11.786 PDT [Consumer >>>>>>>>>>>> clientId=consumer-2, >>>>>>>>>>>> groupId=Reader-0_offset_consumer_1735388161_genericPipe] >>>>>>>>>>>> Resetting offset >>>>>>>>>>>> for partition com.arquivei.dataeng.andre-0 to offset 3. >>>>>>>>>>>> >>>>>>>>>>>> [1]: >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to >>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's >>>>>>>>>>> log >>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to >>>>>>>>>>> avoid it >>>>>>>>>>> without user option. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is >>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi < >>>>>>>>>>>>> [email protected]> escreveu: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for debugging. >>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace >>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which >>>>>>>>>>>>>> can >>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & >>>>>>>>>>>>>> 587 [1]. >>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can >>>>>>>>>>>>>> explain >>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that >>>>>>>>>>>>>> Kafka client >>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as >>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and >>>>>>>>>>>>>> we need to >>>>>>>>>>>>>> restart the client in KafkaIO. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]: >>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Raghu, thanks for your help. >>>>>>>>>>>>>>> Just answering your previous question, the following logs >>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were >>>>>>>>>>>>>>> still getting >>>>>>>>>>>>>>> the messages, for example: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (...) >>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>> to offset 10. >>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>> to offset 15. >>>>>>>>>>>>>>> ERROR >>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>> to offset 22. >>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 >>>>>>>>>>>>>>> to offset 30. >>>>>>>>>>>>>>> (...) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current >>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed >>>>>>>>>>>>>>> message, >>>>>>>>>>>>>>> before the error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a >>>>>>>>>>>>>>> Dev scenario. >>>>>>>>>>>>>>> We started the same pipeline using the direct runner, >>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network >>>>>>>>>>>>>>> and the same >>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the >>>>>>>>>>>>>>> pipeline was able >>>>>>>>>>>>>>> to successfully process the subsequent messages. >>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and >>>>>>>>>>>>>>> did the same test, the same problem from our production >>>>>>>>>>>>>>> scenario happened, >>>>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, >>>>>>>>>>>>>>> we've stopped >>>>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is >>>>>>>>>>>>>>> still >>>>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very >>>>>>>>>>>>>>> much. >>>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi < >>>>>>>>>>>>>>> [email protected]> escreveu: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread >>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one >>>>>>>>>>>>>>>> (if a worker >>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that >>>>>>>>>>>>>>>> KafkaClient itself >>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though >>>>>>>>>>>>>>>> unlikely). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Eduardo, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying >>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you >>>>>>>>>>>>>>>>> see any others >>>>>>>>>>>>>>>>> afterwards in the log? >>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful >>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>> - login to one of the VMs and get stacktrace of java >>>>>>>>>>>>>>>>> worker (look for a container called java-streaming) >>>>>>>>>>>>>>>>> - file a support bug or stackoverflow question with jobid >>>>>>>>>>>>>>>>> so that Dataflow oncall can take a look. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Raghu. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow >>>>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka >>>>>>>>>>>>>>>>>> messages at all, >>>>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get >>>>>>>>>>>>>>>>>> Kafka messages. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] Error >>>>>>>>>>>>>>>>>> sending fetch request (sessionId=1396189203, epoch=2431598) >>>>>>>>>>>>>>>>>> to node 3: >>>>>>>>>>>>>>>>>> org.apache.kafka.common.errors.DisconnectException. >>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] Group >>>>>>>>>>>>>>>>>> coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is >>>>>>>>>>>>>>>>>> unavailable or invalid, will attempt rediscovery >>>>>>>>>>>>>>>>>> I [Consumer clientId=consumer-1, groupId=genericPipe] >>>>>>>>>>>>>>>>>> Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 >>>>>>>>>>>>>>>>>> rack: null) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This is the KafkaIO setup we have: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]() >>>>>>>>>>>>>>>>>> .withBootstrapServers(server) >>>>>>>>>>>>>>>>>> .withTopic(topic) >>>>>>>>>>>>>>>>>> .withKeyDeserializer(classOf[StringDeserializer]) >>>>>>>>>>>>>>>>>> .withValueDeserializer(classOf[StringDeserializer]) >>>>>>>>>>>>>>>>>> .updateConsumerProperties(properties) >>>>>>>>>>>>>>>>>> .commitOffsetsInFinalize() >>>>>>>>>>>>>>>>>> .withoutMetadata() >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Any help will be much appreciated. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria >>>>>>>>>>>>>>>>>> no Vale do Silício] >>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no >>>>>>>>>>>>>>> Vale do Silício] >>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Eduardo Soldera Garcia >>>>>>>>>>>>> Data Engineer >>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no >>>>>>>>>>>>> Vale do Silício] >>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eduardo Soldera Garcia >>>>>>>> Data Engineer >>>>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale >>>>>>>> do Silício] >>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>>>> <https://www.facebook.com/arquivei> >>>>>>>> <https://www.linkedin.com/company/arquivei> >>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Eduardo Soldera Garcia >>>>>> Data Engineer >>>>>> (16) 3509-5555 | www.arquivei.com.br >>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>>>>> Silício] >>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>>> <https://www.facebook.com/arquivei> >>>>>> <https://www.linkedin.com/company/arquivei> >>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Eduardo Soldera Garcia >>>>> Data Engineer >>>>> (16) 3509-5555 | www.arquivei.com.br >>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>>>> Silício] >>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>>>> <https://www.facebook.com/arquivei> >>>>> <https://www.linkedin.com/company/arquivei> >>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>>>> >>>> >>> >>> -- >>> Eduardo Soldera Garcia >>> Data Engineer >>> (16) 3509-5555 | www.arquivei.com.br >>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>> [image: Arquivei.com.br – Inteligência em Notas Fiscais] >>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> >>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do >>> Silício] >>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> >>> <https://www.facebook.com/arquivei> >>> <https://www.linkedin.com/company/arquivei> >>> <https://www.youtube.com/watch?v=sSUUKxbXnxk> >>> >> > > -- > Eduardo Soldera Garcia > Data Engineer > (16) 3509-5555 | www.arquivei.com.br > <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> > [image: Arquivei.com.br – Inteligência em Notas Fiscais] > <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura> > [image: Google seleciona Arquivei para imersão e mentoria no Vale do > Silício] > <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad> > <https://www.facebook.com/arquivei> > <https://www.linkedin.com/company/arquivei> > <https://www.youtube.com/watch?v=sSUUKxbXnxk> >
