Hi Maciek, Thank you for reaching out. I'll try to answer your questions separately.
- nothing comparable. You already mention the State Processor API. Besides that, I can only think of a side channel (CoFunction) that is used to request a certain state that is then send to a side output and ultimate to a sink, e.g. Kafka State Request Topic -> Flink -> Kafka State Response Topic. This puts this complexity into the Flink Job, though. - I think it is a combination of both. Queryable State works well within its limitations. In the case of the RocksDBStatebackend this is mainly the availability of the job and the fact that you might read "uncommitted" state updates. In case of the heap-backed statebackends there are also synchronization issues, e.g. you might read stale values. You also mention the fact that queryable state has been an afterthought when it comes to more recent deployment options. I am not aware of any Committer who currently has the time to work on this to the degree that would be required. So, we thought, it would be more fair and realistic to mark Queryable State as "approaching end of life" in the sense that there is no active development on that component anymore. Best, Konstantin On Tue, Mar 9, 2021 at 7:08 AM Maciek Próchniak <m...@touk.pl> wrote: > Hello, > > > We are using QueryableState in some of Nussknacker deployments as a nice > addition, allowing end users to peek inside job state for a given key > (we mostly use custom operators). > > > Judging by mailing list and feature radar proposition by Stephan: > > https://github.com/StephanEwen/flink-web/blob/feature_radar/img/flink_feature_radar.svg > > > this feature is not widely used/supported. I'd like to ask: > > - are there any alternative ways of accessing state during job > execution? State API is very nice, but it operates on checkpoints and > loading whole state to lookup one key seems a bit heavy? > > - are there any inherent problems in QueryableState design (e.g. it's > not feasible to use it in K8 settings, performance considerations) or > just lack of interest/support (in that case we may offer some help)? > > > thanks, > > maciek > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
