Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
well..major runtime changes are coming with FLIP-6, 15 and 16 so I would
suggest you watch those. Loop FT will be included in one of these along with
other loop redesign features.
---
Github user dikei commented on the issue:
https://github.com/apache/flink/pull/1668
Hi. Do we have any updates on this :)
---
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
sweet! thanks @StefanRRichter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/1668
For raw operator state, override
`AbstractStreamOperator::snapshotState(StateSnapshotContext context)` inside
your operator. Your implementation calls to super, then it can obtain the raw
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Thanks for the review @gyfora and @StephanEwen , these are very good points.
@StephanEwen makes sense to not really index/keep metadata of individual
records in log slices, it is extra
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey. Any update/opinion/something anyone?
Just a gentle reminder, sorry if this sounds a bit desperate :)
---
If your project is set up for it, you can reply to this email and have your
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
I have just rebased to the current master.
Please let me wrap this up. It has been over a year :)
Unregistering state in the OperatorStateStore is very tiny fix.
@StephanEwen
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey @addisonj.
Sure! You could perhaps review the changes and maybe see how to discard
empty operator states if you are motivated. This is the only pending issue for
this PR. thanks!
---
Github user addisonj commented on the issue:
https://github.com/apache/flink/pull/1668
Very interested in this work. It sounds like there are few loose ends and
then some cleanup before it might be ready for merge, @senorcarbone or
@StephanEwen anything that can be supported by
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
The last update implements a variant of what @StephanEwen proposes. We have
put some more thought on this offline too thanks to @gyfora ! The idea is that
instead of putting records to each
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
@StephanEwen could you check my question above?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Working on it atm . I decided to make the following optimisations but want
to very quickly make sure that async checkpointing works the way I believe it
does:
- Most importantly, I am
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
agreed @StephanEwen! I will do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/1668
@senorcarbone I agree, let's fix the multiple checkpoints issue and do the
rest in FLIP-15
The other operators have a pretty simply way of doing this:
- for synchronous
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
These are some good points @StephanEwen, thanks for checking it.
How about the following, regarding each issue:
- `Concurrent Checkpoints`: Looks like an improvement but I can sure
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/1668
To suggest some way to fix the guarantees: To my mind, the crux lies in the
way that the feedback channel is implemented - a simple blocking queue just
does not cut it for that case. To make
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/1668
Thanks for the reminder, I went over the code today. The code looks mostly
good, but here are some thoughts:
- The head task supports only one concurrent checkpoint. In general, the
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Exactly, these two issues do not depend on each other. No doubt loop FT is
the first thing that can enable iterations in a production deployment so I
would merge that first.
Thank you
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/1668
I think the PR looks pretty good, and it sounds fair to to address
termination in a later PR as this will still greatly improve the current
guarantees without making the backpressure/termination
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey again, @StephanEwen @uce .
When you have 10 min can you take a look to see if this is acceptable?
I would not like to leave this here for months again, it has been out way
too long
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Ok, so I am progressing this a bit independently from the termination stuff
and then we rebase to the first PR that is merged. I just changed everything
and rebased to the current master.
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey! Good to be back :) . Let's fix this properly, as @StephanEwen
recommended it now that there is some time.
We are writing together with @FouadMA a FLIP to address major loop fixes.
22 matches
Mail list logo
