Hi Des, Regarding kinesis datastream source: currently it still hasn't supported kinesis source natively in PyFlink DataStream API, however, you could use the Kinesis Table API & SQL connectors [1] and then convert the Table to DataStream [2] if you want to work with PyFlink DataStream API.
Regards, Dian [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kinesis/ [2] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/datastream/intro_to_datastream_api/#create-using-table--sql-connectors On Wed, Nov 10, 2021 at 5:48 PM Desmond F <desfe...@gmail.com> wrote: > Hi all, > > About our use case - > > We have many clients connected via websockets through api gateway on > AWS, these clients submit events of various types periodically, each > event contains a session_id (generated by the client), the session > ends when there's no activity for a specified duration of time. We > have a sequence model (RNN) written in PyTorch that needs to send > predictions back to the clients for each event that is being sent. The > state should contain the raw events sent per session and after a local > transformation it can serve as an input to the model. > > Flink seems like a natural fit for our app (no session store is a > winner) but we are seriously struggling to understand how to implement > it and with which API, currently it boils down to PyFlink or Beam. > > What do you think? > > As I mentioned, our model runs with Python, we are currently trying to > use the PyFlink Datastream API, but now we're not sure whether we can > have a Kinesis data stream source in Python. Is it supported? > > As for the execution environment, we'd like to use a managed service > such as Kinesis Data Analytics, or perhaps we should use EMR with > flink installed... What is recommended? > > Note that we don't care so much about durability and it's ok for our > use-case to lose sessions in case of failures, so with this assumption > we'd like to build the fastest performing setup. > > Appreciate any additional pointers for implementing our app. > > Thanks, > Des. >
