Cool ..Thanks again for you help/suggestions Luke.
Regards
Mohil
On Tue, Jun 2, 2020 at 10:42 AM Luke Cwik wrote:
> Using side inputs is fine and is a common pattern. You should take a look
> at "slowly changing side inputs"[1] as there is some example code there.
>
> 1:
> https://beam.apache.o
Using side inputs is fine and is a common pattern. You should take a look
at "slowly changing side inputs"[1] as there is some example code there.
1:
https://beam.apache.org/documentation/patterns/side-inputs/#slowly-updating-side-input-using-windowing
On Mon, Jun 1, 2020 at 8:27 PM Mohil Khare
Thanks Luke for your reply.
I see. I am trying to recall why I added allowedLateness as 360 days.
Anyways I will try without that.
But do you think the approach I am using to keep getting a running score in
a sliding window and then using it as a side input to decorate the main
log is correct ? O
Your allowed lateness is 360 days and since the trigger you have doesn't
emit speculative results, you'll have to wait till the watermark advances
to the end of windows timestamp + 360 days before something is output from
the grouping aggregation/available at the side input.
On Sat, May 30, 2020
Hello all,
Any suggestions? Where am I going wrong or is there any better way of
achieving this so that I can do replay as well ?
Thanks
Mohil
On Wed, May 27, 2020 at 11:40 AM Mohil Khare wrote:
> Hi everyone,
> I need a suggestion regarding usage of the side input pattern and sliding
> window
Hi everyone,
I need a suggestion regarding usage of the side input pattern and sliding
window, especially while replaying old kafka logs/offsets.
FYI: I am running beam 2.19 on google dataflow.
I have a use case where I read a continuous stream of data from Kafka and
need to calculate one score (
