Hi,
I have the requirement to dedup messages within the window and take bunch of
actions on the filtered message. I understand that we can get parallelism with
the number of Kstream thread and can get maximum parallelism as number of
partitions. But the actions that I take on the filtered message are various IO
operations. As my applications is IO bound, I want to be able to execute
multiple message after the window parallelly, and not sequentially.
I am using following code to start with. The forEach allows to execute messages
sequentially. If I add submit Threads in foreach to thread pool, then I am not
understanding how can I block committing the message until it is successfully
executed. Some message executed in the window can fail as well. Please let me
know, if you have any suggestions to process the windowed messages parallelly
with Kstream.
builder.<String, String>stream(topic)
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMillis(windowedTime)).advanceBy(Duration.ofMillis(windowedTime)))
.reduce((value1, value2) -> value2, Materialized.as(reducerTopic))
.toStream()
.foreach((key, value) -> System.out.println(key + " => " + value));
Thanks,
Divya
