Re: Completeablefuture in a flat map operator

[Lasse Nedergaard]

Hi  In my case I have to query an external system. The system returns n numbers of row in a page and I have to call the system until no more data.  I could use AsyncFunction if I buffer all records and output at the end but I want to “stream” and don’t have enough memory to hold all the data.  AsyncFunction only has one future to return so I can’t see any way to handle n number of results from the external service.  Med venlig hilsen / Best regards Lasse Nedergaard Den 19. feb. 2024 kl. 21.31 skrev Ken Krugler <kkrugler_li...@transpac.com>: Is there some reason why you can’t use an AsyncFunction? AsyncFunction (Flink : 1.20-SNAPSHOT API) nightlies.apache.org <favicon.ico> Note that when dealing with event time and exactly once, an AsyncFunction provides required support for proper execution. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/ for more details. — Ken On Feb 19, 2024, at 12:20 PM, Lasse Nedergaard <lassenedergaardfl...@gmail.com> wrote: Hi. I have a case where I would like to collect object from a completeablefuture future in a flat map function. I run into some problem where I get an exception regarding a buffer pool that don’t exists when I collect the objets after some times.  I can see if I for testing don’t return from the function (creating a fori loop with a thread sleep or wait for the future) it works. Can anyone explain what going on behind the screen and if possible any hints for a working solution. Thanks in advance Med venlig hilsen / Best regards Lasse Nedergaard -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot