Re: AvroUtils converting generic record to Beam Row causes class cast exception

I didn't find code in `AvroUtils.toBeamRowStrict` that converts long to Joda time. `AvroUtils.toBeamRowStrict` retrieves objects from GenericRecord, and tries to cast objects based on their types (and cast(object) to long for "timestamp-millis"). see [1]. So in order to use

Re: AvroUtils converting generic record to Beam Row causes class cast exception

Hi Rui, I agree that by converting it to long, there will be no error. But the KafkaIO is giving a GenericRecord with attribute of type JodaTime. Now I convert it to long. Then in the AvroUtils.toBeamRowStrict again converts it to JodaTime. I used the avro tools 1.8.2 jar, for the below schema

Re: Hi, some sample about Extracting data from Xlsx ?

Hi Pablo , Thanks for your attention, I so sorry, my bad written "Cs extension " I did means .csv extension ! The example like this: load-csv-file-from-google-cloud-storage

Re: Hi, some sample about Extracting data from Xlsx ?

Hello Henrique, I am not aware of existing Beam transforms specifically used for reading in XLSX data. Can you share what you mean by "examples related with Cs extension"? I am aware of some Python libraries foir this sort of thing[1]. You could use the FileIO transforms in the Python SDK to

Hi, some sample about Extracting data from Xlsx ?

Hello I would like to use best practices from Apache Beams to read Xlsx. however I found examples only related with Cs extension. someone there is sample using ParDo to Collect all columns and sheets from Excel xlsx ? Afterwards I will put into google Big query. Thanks & Regards

Re: AvroUtils converting generic record to Beam Row causes class cast exception

Read from the code and seems like as the logical type "timestamp-millis" means, it's expecting millis in Long as values under this logical type. So if you can convert joda-time to millis before calling "AvroUtils.toBeamRowStrict(genericRecord, this.beamSchema)", your exception will gone. -Rui

Re: AvroUtils converting generic record to Beam Row causes class cast exception

+dev On Sun, Apr 14, 2019 at 10:29 PM Vishwas Bm wrote: > Hi, > > Below is my pipeline: > > KafkaSource (KafkaIO.read) --> Pardo ---> BeamSql > ---> KafkaSink(KafkaIO.write) > > > The avro schema of the topic has a field of logical type > timestamp-millis.

Dataflow HA and DR

hi We have a few simple Dataflow Streaming jobs running. Requirement is to build HA/DR solution. a) Is it a good idea to spin multiple Dataflow jobs in different regions listing to same 'shared' pubsub Subscription. b) If not , then can you please share some best practices about it. Thanks