Weird, from experience, I have seen this timeout happen if the specified partition doesn't exist.
However, since you are using Camel 3.9 and you didn't set a message key, the partition key won't be set in the ProducerRecord, it will be only set if you set the message key, however this behavior has changed in Camel 3.10 where the partition key won't require setting the message key. Thus, even if you set the partition key but you didn't set a message key, I will expect camel-kafka to ignore the partition (based on Camel 3.9) as you can see in this code <https://github.com/apache/camel/blob/camel-3.9.0/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java#L275> . How about if you try using camel-vertx-kafka <https://camel.apache.org/components/3.9.x/vertx-kafka-component.html> component and see if this solves the problem? Regards, Omar On Mon, Apr 19, 2021 at 5:26 PM Mark Nuttall <mknutt...@gmail.com> wrote: > Setting the message key does not help. :( > > On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall <mknutt...@gmail.com> wrote: > > > I've never set the message key. I might give that a shot. > > > > The route is pretty simple. And the only headers being set are the > > partition key and another string value i need to be passed. The body is > > JSON. > > I am running Kafka locally (so whatever partitions is the default) and > AWS > > MSK - 2 partitions. > > > > Mark > > > > > > > > On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <o...@oalsafi.com> wrote: > > > >> Hi Mark, > >> > >> When you send your message to Kafka, do you as well set the message key > or > >> you just set the partition key? Perhaps it would be great if you can > post > >> an example for your route with the headers/body that you set, that could > >> help us here to troubleshoot the issue. Also, how many partitions do you > >> have in your topic? > >> > >> Regards, > >> Omar > >> > >> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mknutt...@gmail.com> > wrote: > >> > >> > Camel version: 3.9.0 > >> > > >> > Issue: when consuming from one kafka topic and producing to another, > >> > setting the partition key causes the write to fail. After a few > seconds > >> it > >> > will fail with this error: > >> > > >> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000]. > >> Caused > >> > by: [org.apache.kafka.common.errors.TimeoutException - Topic > some-topic > >> not > >> > present in metadata after 60000 ms.] > >> > > >> > The topic does exist. If i produce to the topic without the partition > >> key > >> > OR I produce to the topic with a partition key via the > producerTemplate > >> it > >> > works just fine. > >> > I am pretty sure in the past i have done this and it worked. > >> > > >> > Has anyone else seen this? Any ideas? I do have a work around (i.e. > use > >> > Spring's kafkaTemplate.). > >> > > >> > Mark > >> > > >> > > >
