1) Outside of small-scale tests the periodic emission of watermarks
should ensure that the latency is not unbounded. Your test just runs so
quickly that this never triggers.
As for the triggering element being emitted first: The reason this
happen is because watermarks are not really attached
Chesnay,
1) Correct, I'd like the timeout event (generated at eventTime==1000) to
appear in its correct time sequence in the output, i.e. before eventTime
exceeds 1000.
It's great that Flink can deal with out-of-orderness, but I didn't expect
it to spontaneously create it (especially with
I'm not sure I see the problem in your output.
For any given key the timestamps are in order, and the events where
devices are offline seem to occur at the right time.
Is it just that you'd like the following line to occur earlier in the
output?
Scratch that - your WatermarkStrategy DOES work (when I implement it
correctly!).
Well, almost: As you can see below (code pushed to repo), the Timer events
are still appearing somewhat late in the stream - 4 events late in this
case. It may be just good-enough for my purposes, though it will make
Chesnay,
I cannot reproduce this - I've tried the approaches you suggest, but
nothing I've done makes the timers fire at the correct time in the stream -
they only fire when the stream has ended. If you have an EventTime example
where they fire at the right time in the stream, I'd love to see it.
Actually, if the parallelism is 1 then it works as it should. sigh
On 1/27/2021 6:52 PM, Chesnay Schepler wrote:
Note that while this does fix the issue of timers not firing while the
job is running, it seems to be firing too many timers...
On 1/27/2021 6:49 PM, Chesnay Schepler wrote:
My
Note that while this does fix the issue of timers not firing while the
job is running, it seems to be firing too many timers...
On 1/27/2021 6:49 PM, Chesnay Schepler wrote:
My bad, I was still using the custom WatermarkStrategy that emits a
watermark for each event.
My bad, I was still using the custom WatermarkStrategy that emits a
watermark for each event.
.assignTimestampsAndWatermarks( new WatermarkStrategy() { @Override
public WatermarkGenerator
createWatermarkGenerator(WatermarkGeneratorSupplier.Context context) {
return new
I am calling
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
I am using Flink 11.1 (because I need to run on AWS Kinesis Data Analytics).
-Pilgrim
--
Learn more at https://devicepilot.com @devicepilot
On 2021/01/27 15:09, Chesnay Schepler wrote:
Put another way, if you use any of the built-in WatermarkGenerators and
use event-time, then it appears that you *must* set this interval.
This behavior is...less than ideal I must admit, and it does not
appear to be properly documented.
Setting
Chesnay,
Thanks for this - I've made the change you suggested
(setAutoWatermarkInterval) but it hasn't changed the behaviour - timers
still get processed only on stream end.
I have pushed a new version, with this change, and also emitting some
information in a .log field.
If you search for "!!!"
||
You were right that it is an issue with the watermarks; outside of the
when the job was stopped they were never emitted downstream, so no timer
was ever triggered.
It appears that you need to set the setAutoWatermarkInterval in the
ExecutionConfig via
Based on your description you aren't doing anything obviously wrong.
Would it be possible for you to share the code with us?
On 1/27/2021 1:02 PM, Pilgrim Beart wrote:
A newbie question:
I've created a basic Flink DataStream job for an IoT use-case, with
file source and sink for testing.
I
A newbie question:
I've created a basic Flink DataStream job for an IoT use-case, with file
source and sink for testing.
I key by device ID, then in a ProcessFunction set an EventTime Timer to
fire if a device falls silent, i.e. a timeout, which I cancel if another
message arrives from that
14 matches
Mail list logo
