Re: How to integrate Beam SQL windowing query with KafkaIO?

Hi Rui, Thanks for your advice! After reading Chapter 2&3 of *Streaming Systems* and some other materials, eventually I make it work! It indeed turned out to be an issue of not setting the trigger correctly. Previously, I didn't set the trigger & watermark so it would use the default settings. Af

Re: [Java - Beam Schema] Manually Generating a Beam Schema for a POJO class

Hi Brian, Thank you for opening the issue! My current workaround is to generate a BigQuery schema with helper functions I already have (since I am writing to BigQuery in the end in my sink). I have the Beam Schema function still in the code, but I currently don't use them as I couldn't make them w

Is KafkaIO KafkaWriter stateless

Hi all, Just would like to confirm, KafkaWriter has no state, so that means, when enabled checkpoint, no state will be checkpointed from KafkaWriter? Thanks a lot! Eleanore

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Kyle, Thanks for the response! On Wed, Aug 26, 2020 at 5:28 PM Kyle Weaver wrote: > > - With the Flink operator, I was able to submit a Beam job, but hit the > issue that I need Docker installed on my Flink nodes. I haven't yet tried > changing the operator's yaml files to add Docker inside

Re: How to integrate Beam SQL windowing query with KafkaIO?

Glad it has worked! So sounds like data has been dropped as they are considered late data and `.withAllowedLateness()` make the data emitted. -Rui On Thu, Aug 27, 2020 at 10:09 AM Minreng Wu wrote: > Hi Rui, > > Thanks for your advice! > > After reading Chapter 2&3 of *Streaming Systems* and

Handshake_failure running a Dataflow pipeline

Hello, Apache Beam community! Hope everything goes ok at this time. I write to you asking for your guide and help. I have been facing some problems accessing HTTPS resources from a pipeline deployed in Dataflow. The problem occurs only when I run with DataflowRunner, the DirectRunner is working o

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Eugene! I’m struggling to find complete documentation on how to do this. There seems to be lots of conflicting or incomplete information: several ways to deploy Flink, several ways to get Beam working with it, bizarre StackOverflow questions, and no documentation explaining a complete working e