Re: Question on late data handling in Beam streaming mode

Thanks folks. This is really informative! From: Kenneth Knowles Reply-To: "user@beam.apache.org" Date: Friday, April 23, 2021 at 9:34 AM To: Reuven Lax Cc: user , Kenneth Knowles , Kelly Smith , Lian Jiang Subject: Re: Question on late data handling in Beam streaming mode Reuve

Re: Question on late data handling in Beam streaming mode

atever window covers its arrival time and thus that late >> data will not get lost. >> >> >> >> Does Beam support this kind of mechanism? Thanks. >> >> >> >> *From: *Kenneth Knowles >> *Reply-To: *"user@beam.apache.org" &g

Re: Question on late data handling in Beam streaming mode

Beam support this kind of mechanism? Thanks. > > > > *From: *Kenneth Knowles > *Reply-To: *"user@beam.apache.org" > *Date: *Thursday, April 22, 2021 at 1:49 PM > *To: *user > *Cc: *Kelly Smith , Lian Jiang < > li...@zillowgroup.com> > *Subject: *Re: Q

Re: Question on late data handling in Beam streaming mode

s kind of mechanism? Thanks. From: Kenneth Knowles Reply-To: "user@beam.apache.org" Date: Thursday, April 22, 2021 at 1:49 PM To: user Cc: Kelly Smith , Lian Jiang Subject: Re: Question on late data handling in Beam streaming mode Hello! In a streaming app, you have two choices:

Re: Question on late data handling in Beam streaming mode

Hello! In a streaming app, you have two choices: wait forever and never have any output OR use some method to decide that aggregation is "done". In Beam, the way you decide that aggregation is "done" is the watermark. When the watermark predicts no more data for an aggregation, then the aggregati

Question on late data handling in Beam streaming mode

Hi Beam community, I am wondering if there is a risk of losing late data from a Beam stream app due to watermarking? I just went through this design doc and noticed the “droppable” definition there: https://docs.google.com/document/d/12r7frmxNickxB5tbpuEh_n35_IJeVZn1peOrBrhhP6Y/edit# Can you