| Hi
In my case I have to query an external system. The system returns n numbers of row in a page and I have to call the system until no more data. I could use AsyncFunction if I buffer all records and output at the end but I want to “stream” and don’t have enough memory to hold all the data. AsyncFunction only has one future to return so I can’t see any way to handle n number of results from the external service. Med venlig hilsen / Best regards Lasse Nedergaard
Den 19. feb. 2024 kl. 21.31 skrev Ken Krugler <kkrugler_li...@transpac.com>:
Is there some reason why you can’t use an AsyncFunction?
Note that when dealing with event time and exactly once, an AsyncFunction provides required support for proper execution.
See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/ for more details.
— Ken
On Feb 19, 2024, at 12:20 PM, Lasse Nedergaard <lassenedergaardfl...@gmail.com> wrote:
Hi.
I have a case where I would like to collect object from a completeablefuture future in a flat map function.
I run into some problem where I get an exception regarding a buffer pool that don’t exists when I collect the objets after some times. I can see if I for testing don’t return from the function (creating a fori loop with a thread sleep or wait for the future) it works.
Can anyone explain what going on behind the screen and if possible any hints for a working solution.
Thanks in advance
Med venlig hilsen / Best regards
Lasse Nedergaard
-------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot
|
