Re: [Question] Is message order preserved in windows?

Hi Gyorgy, To my knowledge, ordering of elements is not guaranteed. To order your elements in each window by time, you would need to sort the iterables produced by the groupByKey. You might use SortValues[1] to do that. [1] https://beam.apache.org/documentation/sdks/java-extensions/#example-usage

Writing out results to MySQL database

Hi, We have a datapipeline that produces ~400M datapoints each day. If we run it without storing, it finishes in a little over an hour. If we run it and store the datapoints in a MySQL database it takes several hours. We are running on GCP dataflow, the MySQL instances are hosted GCP instances. W

[Question] Route late data to another output

Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in re

Re: [Question] - Side inputs in golang sdk

Ah. And in re-reading your email, that also sounds like a bug we can fix. Unbounded PCollections are quite new to the Go SDK, PubSub on Dataflow being the only way to produce them for a long time. There's bound to be a few of those . A glance at the handling code doesn't indicate there should be

Re: [Question] - Side inputs in golang sdk

Hello Kane! In chatting with Jack, this won't be possible to implement in the Go SDK until 2.40. DoFn self checkpointing only made it into the Go SDK a week or two ago. That is required for unbounded PCollections. A Go native Periodic Sequence isn't yet implemented at head, but there shouldn't be

Re: [Question] - Side inputs in golang sdk

Hi Jack, Thanks for the quick reply. Having tried to add windowing that does not seem to help. Here is an example pipeline [1]. I'm trying to re-create the slowly updating side-input outlined in [2], a common beam pattern. The side-input is created as a bounded node in the graph, and it is not

[Question] Is message order preserved in windows?

Hi, I have video analytic data: - time - object type (car, person, etc) - object bounding box (x,y,width,height) - track id I'd like to do transformations where I need to process detections belonging to the same track in time order. My input data is ordered by time. I do some filtering first (eg