Hi Kafka is a lot more pain to use as a client than traditional broker systems that has clients that handle a lot more reconnectivity and whatnot.
So to make kafka client fail or detect that a topic is not present on the broker, then its subscribe method successed anyway, and you would need to check yourself if any topics on the broker has a similar name as the one you want to consume from I created a ticket https://issues.apache.org/jira/browse/CAMEL-21774 If you want to let Camel routing engine able to deal with this then this is not possible as the subscription happens before the polling. And the kafka poll method inside Kafka does not propagate the fact that the topic is unknown. There is no way to find out. So we may need to add a special option to turn on a a check (overhead) that if there are no records then it may due to unknown topic. But that is another network call to kafka On Fri, Jan 31, 2025 at 3:23 PM Claus Ibsen <claus.ib...@gmail.com> wrote: > Hi > > > For Camel in general there is docs on the website, books, videos on > you-tube and so forth. > > This is a volunteer based community, and a bunch of core engineers that > are very busy. > I am afraid we don't have excess time to help when its too general and > environment specific questions/ > > There are commercial companies you can contract to get assistance > https://camel.apache.org/manual/commercial-camel-offerings.html > > > On Fri, Jan 31, 2025 at 2:49 PM Vineet Kumar > <vineet.ku...@iongroup.com.invalid> wrote: > >> Hello Team, >> >> I am reaching out on behalf of ION Group regarding some issues we've >> encountered while working with Apache Camel in our Kafka-based projects. We >> are seeking your assistance to resolve some challenges related to exception >> handling in both producer and consumer routes. We have been working with a >> Kafka producer and consumer route in Camel, and our goal is to catch >> exceptions if something goes wrong in the Kafka producer or consumer routes. >> >> >> *Project 1: *Latest Version of Libraries (Camel: 4.8.0, Kafka Clients: >> 3.9.0) - Template POC Project >> *Findings:* >> >> - *Producer Route:* We successfully implemented exception handling >> for the producer route by adding onException block on global level with >> maxBlockMs=2500 config. >> - *This allowed us to catch exceptions within 2.5 seconds if any >> error occurred during polling, which triggered the OnException block.* >> - *Consumer Route:* Unfortunately, we were unable to achieve >> similar results with the consumer route. Even when reproducing an >> exception >> (e.g., *UnknownTopicOrPartitionException*) >> - *The consumer route continued polling indefinitely, and the >> OnException block was never triggered.* >> - While we found that we could capture exceptions like * >> UnknownTopicOrPartitionException *for the producer route, the same >> did not apply to the consumer route. >> - There doesn't seem to be a similar config (like maxBlockMs) for the >> consumer route, and we're unable to handle exceptions in the same way. >> - We were only able to catch >> *org.apache.kafka.common.errors.TimeoutException >> **on which we invoked getCause method to get root cause wrapped >> exception** i.e, UnknownTopicOrPartitionException* >> >> >> >> *Project 2: *Older Version of Libraries (Camel: 3.14.10, Kafka Clients: >> 2.7.2) - *Actual version we want to use* >> *Findings:* >> >> - *Producer Route:* We were able to reproduce a timeout exception, >> but when calling .getCause() on the Throwable object, we received a >> null result. >> - In this version, only the TimeoutException was captured, and the >> root cause was not available via getCause method invocation. >> - *Consumer Route:* Similarly, we were unable to catch any >> exceptions in the consumer route, which resulted in indefinite polling. >> - We used similar example like producer: We tried consuming from a >> topic which doesn't exist on the kafka broker >> - Note:* auto.create.topics.enable=false* was added explicitly in >> server.properties file used by kafka broker to prevent auto >> creation of >> topic >> >> >> >> *Steps We've Taken:* To resolve these issues, we've tried the following >> below approaches in both versions and none of them seems to work out for us >> except onException for producer just for TimeoutException. >> >> 1. *OnException: Only caught timeout exceptions in the producer route >> (Only timeout exceptions)* >> 2. *ErrorHandler: We tested both custom error and exception handlers >> without success.* >> 3. *PollExceptionStrategy* >> 4. *BreakOnFirstError* >> 5. *PollOnError* >> 6. *BridgeErrorHandler* >> 7. *ExceptionHandler* >> >> >> >> *Questions:* We would appreciate your guidance on the following: >> >> 1. *Are we missing any key configurations or settings that could help >> resolve this issue to catch any type of exception defined in Errors.class >> file under kafka-clients library?* >> 1. *Includes both retriable and non-retriable exceptions* >> 2. *Could there be an error in our setup or configuration?* >> 3. *Is there any relevant information missing from the Camel >> documentation that could help us handle these exceptions properly?* >> 4. *Is this a known issue or a potential bug in the Camel version >> we're using in both projects?* >> 5. *Any other suggestions or insights you may have would be greatly >> appreciated.* >> >> >> For your reference, I've attached the project zip file on the below Zulip >> chat link containing relevant information, screenshot including the >> dependency version details and all necessary project files. >> Zulip chat link: (46) #general > Assistance Required on Exception >> Handling in Camel Kafka Con - camel - Zulip >> <https://camel.zulipchat.com/#narrow/channel/257295-general/topic/Assistance.20Required.20on.20Exception.20Handling.20in.20Camel.20Kafka.20Con> >> Thank you in advance for your time and support. We look forward to your >> guidance on this matter. >> >> >> Regards, >> [image: ION] >> *Vineet Kumar* >> vineet.ku...@iongroup.com >> *m* >> +91 9003801369 >> iongroup.com <https://www.iongroup.com/> >> >> [image: Impact] >> <https://www.linkedin.com/showcase/ioncommodities/posts/?feedView=all> >> >> > > -- > Claus Ibsen > ----------------- > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > -- Claus Ibsen ----------------- @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
