Re: Statefun embedded functions - parallel per partition, sequential per key

2021-11-03 Thread Igal Shilman
Glad to hear it worked out for you :-) Cheers, Igal On Tue, Nov 2, 2021 at 1:57 PM Filip Karnicki wrote: > Hi All > > Just an update for future reference, it turned out that the machine we > were using for this test didn't have enough memory for what we were asking > it to do. It was that

Re: Statefun embedded functions - parallel per partition, sequential per key

2021-11-02 Thread Filip Karnicki
Hi All Just an update for future reference, it turned out that the machine we were using for this test didn't have enough memory for what we were asking it to do. It was that simple. The upside is that not even with the world's most unstable cluster did we manage to lose a single message. Just

Re: Statefun embedded functions - parallel per partition, sequential per key

2021-10-27 Thread Filip Karnicki
Thanks for your reply Igal The reason why I'm using data stream integration is that the messages on kafka are in .json, and I need to convert them to protobufs for embedded functions. If I was using remote functions I wouldn't need to do that. With regards to performance, in order to exclude the

Re: Statefun embedded functions - parallel per partition, sequential per key

2021-10-27 Thread Igal Shilman
Hello Fil, Indeed what you are describing is exactly what a remote function does. I am curious to learn more about the current performance limitations that you encounter with the remote functions. One thing to try in combination with the async transport, is to increase the total number of in

Statefun embedded functions - parallel per partition, sequential per key

2021-10-27 Thread Filip Karnicki
Hi I have a kafka topic with json messages that I map to protobufs within a data stream, and then send those to embedded stateful functions using the datastream integration api (DataStream[RoutableMessage]). From there I need to make an idempotent long-running blocking IO call. I noticed that I