Yes Gordon, it's obviously gave me a starting point to think about. On Wed, Feb 3, 2021, 12:02 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote:
> Hi, > > There is no out-of-box Flink source/sink connector for this, but it isn't > unheard of that users have implemented something to support what you > outlined. > > One way to possibly achieve this is: in terms of a Flink streaming job > graph, what you would need to do is co-locate the source (which exposes the > endpoint and maintains a pool of open client connections mapped by request > ID), and the sink operators (which receives processed results with the > original request IDs attached, and is in charge for replying to the > original > requests). The open client connections need to be process-wide accessible > (e.g. via a static reference), so that when a co-located sink operator > receives a result, it can directly fetch the corresponding client > connection > and return a response. > > The specifics are of course a bit more evolved; probably need some digging > around previous Flink Forward conference talks to get a better picture. > Hopefully this gives you a starting point to think about. > > Cheers, > Gordon > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
