Re: Apache BEAM on Flink in production

Hi Stephen, There is another relevant video from GCP Next on where Lyft presented their use case, together with some really cool Dataflow announcements from Sergei , Dataflow PM @Google. Regards. *From: *Maximilian Mic

Re: Question about unbounded in-memory PCollection

Hi Rui, Thanks for your reply. This is what I want!! For TestStream.java [1], my understanding is I can attach an event time to my record and let watermark advance to get unbounded

Re: How to configure TLS connections for Dataflow job that use Kafka and Schema Registry

Thanks, I explicitly set file location and password in JAVA_TOOL_OPTIONS and Consumer Factory. Consumer Factory downloads keyStore and trusStore from GCS to local as follows. https://stackoverflow.com/questions/42726011/truststore-and-google-cloud-dataflow?noredirect=1&lq=1 Then I got FileNotExec

Re: How to configure TLS connections for Dataflow job that use Kafka and Schema Registry

Hi Yohei, The configuration exposed as environment variable was picked up by the Beam java program. If you check the backtrace of the error, it has method call to confluent schemaRegistry during deserialization of records io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpReques

Re: How to configure TLS connections for Dataflow job that use Kafka and Schema Registry

Thanks. Did you use this configurations in your Apache Beam application? Does Apache Beam use Confluent Schema Registry client internally ? Yohei Onishi On Thu, May 9, 2019 at 1:12 PM Vishwas Bm wrote: > Hi Yohei, > > I had tried some time back with direct-runner and faced the same issue as >