Re: Consume Kafka messages in a batch

2023-03-22 Thread Mark Nuttall
Thanks Otavio! Option 1 won't work because the Camel code, while it does consume a batch based on those settings and iterates over the enumeration, it only ever passes one at a time. I thought about hacking that class but when we update camel versions, i'd have to remember i did that. :) . We can

Re: Consume Kafka messages in a batch

2023-03-22 Thread Otavio Rodolfo Piske
Hi, Some ideas worth investigating: 1. Using a mix of current kafka client options (i.e.: like maxPollrecords, maxPollInterval, etc) along with aggregate EIP 2. Create your own KafkaClientFactory that wraps a custom Producer/Consumer wrapping the Spring Kafka Consumer. If none of this works, the

Consume Kafka messages in a batch

2023-03-15 Thread Mark Nuttall
I know consuming Kafka messages in a batch is not currently supported. I googled and I didn't find any real options. I looked at hacking the Camel classes to implement it and decided that it was too risky. So for now we are falling back to using a Spring Kafka Consumer. The issue with that is I s