> running a batch or "bounded stream" job first to generate a "cache state",
> and then launching the main streaming job, which loads this initial state
> load in open()... not sure how to work out the keying.
>
This is the recommended workaround this issue - first start a job to precompute
Thanks for the thoughts Piotr.
Seems I have a talent for asking (nearly) the same question as
someone else at the same time, and the check-pointing was raised
in that thread as well.
I guess one way to conceptualize it is that you have is a stream
job that has
Maybe it could work with Flink’s 1.5 credit base flow control. But you would
need a way to express state “block one input side of the CoProcessFunction”,
pass this information up to the input gate and handle it probably similar to
how `org.apache.flink.streaming.runtime.io.CachedBufferBlocker`
I was just thinking about about letting a coprocessfunction "block"
or cause back pressure on one of it's streams?
Has this been discussed as an option?
Does anyone know a way to effectively accomplish this?
I think I could get a lot of mileage out of something
