Re: Using a ProcessFunction as a "Source"

As an update, there is now also this FLIP for the source refactoring: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > On 1. Nov 2018, at 20:47, Addison Higham wrote: > > This is fairly stale, but getting back to this: > > We ended up going the route of

Re: Using a ProcessFunction as a "Source"

This is fairly stale, but getting back to this: We ended up going the route of using the Operator API and implementing something similar to the `readFile` API with one real source function that reads out splits and then a small abstraction over the AbstractStreamOperator, a `MessagableSourceFunct

Re: Using a ProcessFunction as a "Source"

Hi Addison, for a while now different ideas about reworking the Source interface have been floated. I implemented a prototype that showcases my favoured approach for such a new interface: https://github.com/aljoscha/flink/commits/refactor-source-interface

Re: Using a ProcessFunction as a "Source"

The SourceFunction interface is rather flexible so you can do pretty much whatever you want. Exact implementation depends on whether control messages are pulled or pushed to the source; in the first case you'd simply block within "run()" on the external call, in the latter you'd have it block o

Re: Using a ProcessFunction as a "Source"

Hi Addison, I have a lot of things I don't understand. Is your source self-generated message? Why can't source receive input? If the source is unacceptable then why is it called source? Isn't kafka-connector the input as source? If you mean that under normal circumstances it can't receive another

Using a ProcessFunction as a "Source"

HI, I am writing a parallel source function that ideally needs to receive some messages as control information (specifically, a state message on where to start reading from a kinesis stream). As far as I can tell, there isn't a way to make a sourceFunction receive input (which makes sense) so I am