Re: Far too few watermarks getting generated with Kafka source

ason for the split is that the data used to come from two different >> topics). >> >> William >> >> >> >> >> - Original Message - >> From: >> "Gary Yao" >> >> To: >> "William Saar"

Re: Far too few watermarks getting generated with Kafka source

> From: > "Gary Yao" > > To: > "William Saar" > Cc: > "user" > Sent: > Thu, 18 Jan 2018 11:11:17 +0100 > Subject: > Re: Far too few watermarks getting generated with Kafka source > > > > Hi William, > > How often d

Re: Far too few watermarks getting generated with Kafka source

data used to come from two different topics). William - Original Message - From: "Gary Yao" To: "William Saar" Cc: "user" Sent: Thu, 18 Jan 2018 11:11:17 +0100 Subject: Re: Far too few watermarks getting generated with Kafka source Hi William, How often

Re: Far too few watermarks getting generated with Kafka source

Hi William, How often does the Watermark get updated? Can you share your code that generates the watermarks? Watermarks should be strictly ascending. If your code produces watermarks that are not ascending, smaller ones will be discarded. Could it be that the events in Kafka are more "out of order

Far too few watermarks getting generated with Kafka source

Hi, I have a job where we read data from either Kafka or a file (for testing), decode the entries and flat map them into events, and then add a timestamp and watermark assigner to the events in a later operation. This seems to generate periodic watermarks when running from a file, but when Kafka is