Thanks Quingsheng, that would definitely work. But I'm unable to figure out how I can apply this with CoProcessFunction. One stream is windowed and trigger implementation uses the 2nd stream.
On Mon, Jun 27, 2022 at 3:29 PM Qingsheng Ren <re...@apache.org> wrote: > Hi Gopi, > > What about using a window with a custom trigger? The window is doing > nothing but aggregating your input to a collection. The trigger accepts > metadata from the low input stream so it can fire and purge the window > (emit all elements in the window to downstream) on arrival of metadata. > > Best, > Qingsheng > > > On Jun 27, 2022, at 12:46, Gopi Krishna M <gopikrish...@gmail.com> > wrote: > > > > Hi, > > I've a scenario where I use connected streams where one is a low > throughput metadata stream and another one is a high throughput data > stream. I use CoProcessFunction that operates on a data stream with > behavior controlled by a metadata stream. > > > > Is there a way to slow down/pause the high throughput data stream until > I've received one entry from the metadata stream? It's possible that by the > time I get the first element from the metadata stream, I might get 1000s of > items from the data stream. One option is to create a state to buffer the > data stream within the operator. Is there any other option which doesn't > need this state management? > > > > Thanks, > > Gopi > >
