Hi Stephan,
I think that what you are trying to achieve is very interesting, and
possibly other users might find that useful as well
and we will definitely add that to our roadmap.
I think that Gordon's suggestion of using the state processor API to
examine a savepoint, makes a lot of sense in this case.
A savepoint can be analyzed off-band with the DataSet API and sliced and
diced however you'd like. I think that
This approach will also be a major part of the solution we will go with. If
you are interested to work on a contribution here, me and Gordon
would be more than happy to guide you toward that.
Meanwhile, your suggestion to stream the changes to a temporary storage
that is optimized for drill down exploration might also work.
I'd like to suggest an alternative/complementing approach - how about
implementing these requirements in StateFun itself?
First I'd like to refer you to a blog post at [2] and in particular the
image at the bottom, perhaps it can also work in your case.
In your case, you will need an additional function type that will hold the
aggregation data (possibly sharded to reduce hotspots).
The aggregation function can periodically emit the aggregations elsewhere.
To implement a drill down functionality you would have to implement sort-of
an RPC mechanism with your functions (see here for example [3][4]) to ask
a specific function instance for its state.
[1] https://flink.apache.org/feature/2019/09/13/state-processor-api.html
[2] https://flink.apache.org/2020/08/19/statefun.html
[3]
https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-protocol/src/main/protobuf/ridesharing.proto
[4]
https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-example-functions/src/main/java/org/apache/flink/statefun/examples/ridesharing/Module.java
Kind regards,
Igal.
On Thu, Jan 28, 2021 at 7:45 PM Stephan Pelikan
wrote:
> Hi Gordon,
>
>
>
> If operating on checkpoints instead of savepoints this might be OK. But
> since this is not in the current scope I digged into Flink docs and found
> the "queryable state" (
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/queryable_state.html#querying-state).
>
>
>
>
> This sounds good and seems to be a possibility to read the state of a
> specific function by id. This would solve the first part of my challange
> (examining the current state). Additionally there is remote client what
> makes things easy.
>
>
>
> As far as I understand its only necessary to enable this for statefuns. If
> the types like PersistedValue also takes a queryable-name like
> ValueStateDescriptor it could be passed through in places like
> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/state/FlinkState.java#L65.
> Then the state of single jobs could be retrieved if I'm right. But I can
> only query states of a specific statefun by id. Not the total crowd of
> states.
>
>
>
> To get a solution in the "near" future I could send "state changes" egress
> messages and stream them into an ElasticSearch sink. Then I could search
> that ES index the way I like. I only have to check if that works in terms
> of amount of data and throughput. Additionally I'll have to consider how to
> structure those "state changes" events in the ES to be able to query as I
> need. As a give-away I would get historical data of states outdated or
> cleared.
>
>
>
> This sounds like a feasible solution. What do you think?
>
>
>
> Cheers,
>
> Stephan
>
>
>
>
>
> *Von:* Tzu-Li (Gordon) Tai
> *Gesendet:* Donnerstag, 28. Jänner 2021 04:06
> *An:* Stephan Pelikan
> *Cc:* user@flink.apache.org
> *Betreff:* Re: Stateful Functions - accessing the state aside of normal
> processing
>
>
>
> Hi Stephan,
>
> Great to hear about your experience with StateFun so far!
>
> I think what you are looking for is a way to read StateFun checkpoints,
> which are basically an immutable consistent point-in-time snapshot of all
> the states across all your functions, and run some computation or simply to
> explore the state values.
> StateFun checkpoints are essentially adopted from Flink, so you can find
> more detail about that here [1].
>
> Currently, StateFun does provide a means for state "bootstrapping":
> running a batch offline job to write and compose a StateFun checkpoint [2].
> What is still missing is the "reading / analysis" side of things, to do
> exactly what you described: running a separate batch offline job for
> reading and processing an existing St
