Sorry for the delayed response but I'm glad to hear you have solved the
problem.
Piotrek
czw., 24 cze 2021 o 10:55 Felipe Gutierrez
napisał(a):
> So, just an update.
>
> When I used this code (My stateful watermark) on the original application
> it seems that I can recover the latest
So, just an update.
When I used this code (My stateful watermark) on the original application
it seems that I can recover the latest watermark and further process the
join with stuck events on it.
I don't even have to create MyCoProcessFunction to implement a low-level
join. The available
Hello Piotr,
Could you please help me to ensure that I am implementing it in the correct
way?
I created the WatermarkFunction [1] based on the FilterFunction from Flink
and the WatermarkStreamOperator [2] and I am doing unit test [3]. Then
there are things that I am not sure how to do.
How to
I'm glad I could help, I hope it will solve your problem :)
Best,
Piotrek
pt., 18 cze 2021 o 14:38 Felipe Gutierrez
napisał(a):
> On Fri, Jun 18, 2021 at 1:41 PM Piotr Nowojski
> wrote:
>
>> Hi,
>>
>> Keep in mind that this is a quite low level approach to this problem. It
>> would be much
On Fri, Jun 18, 2021 at 1:41 PM Piotr Nowojski wrote:
> Hi,
>
> Keep in mind that this is a quite low level approach to this problem. It
> would be much better to make sure that after recovery watermarks are still
> being emitted.
>
yes. Indeed it looks like a very low level. I did a small test
Hi,
Keep in mind that this is a quite low level approach to this problem. It
would be much better to make sure that after recovery watermarks are still
being emitted.
If you are using a built-in source, it's probably easier to do it in a
custom operator. I would try to implement a custom one
Hello Piotrek,
On Fri, Jun 18, 2021 at 11:48 AM Piotr Nowojski
wrote:
> Hi,
>
> As far as I can tell timers should be checkpointed and recovered. What may
> be happening is that the state of the last seen watermarks by operators on
> different inputs and different channels inside an input is
Hi,
As far as I can tell timers should be checkpointed and recovered. What may
be happening is that the state of the last seen watermarks by operators on
different inputs and different channels inside an input is not persisted.
Flink is assuming that after the restart, watermark assigners will
Hi community,
I have implemented a join function using CoProcessFunction with
CheckpointedFunction to recover from failures. I added some debug lines to
check if it is restoring and it does. Before the crash, I process events
that fall at processElement2. I create snapshots at snapshotState(),
