Re: kafka consumer to kafka producer parition key issue

2021-04-19 Thread Mark Nuttall
Thanks Omar. I was looking at ProducerRecord because I was thinking about trying to use Spring Kafka. I am not sure yet, but part of the issue is that in the camel code, what is being called the paritionKey is not. It is the partition (number). The docs do the same thing. The partitionKey is a val

Re: kafka consumer to kafka producer parition key issue

2021-04-19 Thread Omar Al-Safi
Weird, from experience, I have seen this timeout happen if the specified partition doesn't exist. However, since you are using Camel 3.9 and you didn't set a message key, the partition key won't be set in the ProducerRecord, it will be only set if you set the message key, however this behavior has

Re: kafka consumer to kafka producer parition key issue

2021-04-19 Thread Mark Nuttall
Setting the message key does not help. :( On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall wrote: > I've never set the message key. I might give that a shot. > > The route is pretty simple. And the only headers being set are the > partition key and another string value i need to be passed. The bod

Re: kafka consumer to kafka producer parition key issue

2021-04-19 Thread Mark Nuttall
I've never set the message key. I might give that a shot. The route is pretty simple. And the only headers being set are the partition key and another string value i need to be passed. The body is JSON. I am running Kafka locally (so whatever partitions is the default) and AWS MSK - 2 partitions

Re: kafka consumer to kafka producer parition key issue

2021-04-19 Thread Omar Al-Safi
Hi Mark, When you send your message to Kafka, do you as well set the message key or you just set the partition key? Perhaps it would be great if you can post an example for your route with the headers/body that you set, that could help us here to troubleshoot the issue. Also, how many partitions d

kafka consumer to kafka producer parition key issue

2021-04-16 Thread Mark Nuttall
Camel version: 3.9.0 Issue: when consuming from one kafka topic and producing to another, setting the partition key causes the write to fail. After a few seconds it will fail with this error: Error during processing. Exchange[2729AA824AE4E97-]. Caused by: [org.apache.kafka.common.