I haven't been able to get WatermarkStrategy.withIdleness to work. Is it broken? None of my timers trigger when I'd expect idleness to take over.
On Tue, Mar 2, 2021 at 11:15 PM Dan Hill <quietgol...@gmail.com> wrote: > Hi. > > For local and tests development, I want to flush the events in my system > to make sure I'm processing everything. My watermark does not progress to > finish all of the data. > > What's the best practice for local development or tests? > > If I use idle sources for 1 Kafka partition, this appears broken. I'm > guessing there is logic to prevent removing an idle partition if it's the > only partition. Is there a version of this I can enable for local > development that supports 1 partition? > > I see this tech talk. Are there other talks to watch? > https://www.youtube.com/watch?v=bQmz7JOmE_4&feature=youtu.be > > Do I need to write my own watermark generator? Or change my test data to > have a way of generating watermarks? > > I've tried a few variants of the following source code. The watermark > doesn't progress in the operator right after creating the source. > > SingleOutputStreamOperator<T> viewInput = env.addSource(...) > .uid("source-view") > .assignTimestampsAndWatermarks( > > WatermarkStrategy.<T>forBoundedOutOfOrderness(Duration.ofSeconds(1)).withIdleness(Duration.ofMinutes(1))); >
