Hi Eleanore, Maybe batched RPC is what you are looking for? https://beam.apache.org/blog/timely-processing/
On Wed, Jul 8, 2020 at 6:20 PM Eleanore Jin <eleanore....@gmail.com> wrote: > Thanks Luke and Max for the information. > > We have the use case that inside a DoFn, we will need to call external > services to trigger some other flows. The calls to other services are REST > based sync calls, and it will take 150 milliseconds plus to return. We are > using Flink as the runner and I came across this Async I/O operator from > flink, trying to figure out if this is the right approach and if Beam > provides any similar concept for it. > > Thanks! > Eleanore > > On Wed, Jul 8, 2020 at 2:55 AM Maximilian Michels <m...@apache.org> wrote: > >> Just to clarify: We could make the AsnycIO operator also available in >> Beam but the operator has to be represented by a concept in Beam. >> Otherwise, there is no way to know when to produce it as part of the >> translation. >> >> On 08.07.20 11:53, Maximilian Michels wrote: >> > Flink's AsycIO operator is useful for processing io-bound operations, >> > e.g. sending network requests. Like Luke mentioned, it is not available >> > in Beam. >> > >> > -Max >> > >> > On 07.07.20 22:11, Luke Cwik wrote: >> >> Beam is a layer that sits on top of execution engines like Flink and >> >> provides its own programming model thus native operators like Flink's >> >> async IO operator are not exposed. >> >> >> >> Most people use a DoFn to do all their IO and sometimes will compose >> >> it with another transform such as GroupIntoBatches[1] to simplify >> >> their implementation. >> >> >> >> Why do you need async? >> >> >> >> 1: >> >> >> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/ >> >> >> >> >> >> >> >> On Tue, Jul 7, 2020 at 11:03 AM Eleanore Jin <eleanore....@gmail.com >> >> <mailto:eleanore....@gmail.com>> wrote: >> >> >> >> Hi community, >> >> >> >> I cannot find any documentation for Beam supporting Flink async IO >> >> operator >> >> >> >> ( >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html), >> >> >> >> >> just wonder is this not supported right now? >> >> >> >> Thanks a lot! >> >> Eleanore >> >> >> >
