Hi Okay so we detect this problem when subscribing specially and now also support letting Camel onException be triggered if you enable bridgeErrorHandler=true. This will be in Camel 4.11 onwards.
On Fri, Feb 21, 2025 at 1:51 PM Claus Ibsen <claus.ib...@gmail.com> wrote: > Hi > > Kafka is a lot more pain to use as a client than traditional broker > systems that has clients that handle a lot more reconnectivity and whatnot. > > So to make kafka client fail or detect that a topic is not present on the > broker, then its subscribe method successed anyway, and you would need to > check > yourself if any topics on the broker has a similar name as the one you > want to consume from > > I created a ticket > https://issues.apache.org/jira/browse/CAMEL-21774 > > If you want to let Camel routing engine able to deal with this then this > is not possible as the subscription happens before the polling. > And the kafka poll method inside Kafka does not propagate the fact that > the topic is unknown. There is no way to find out. > > So we may need to add a special option to turn on a a check (overhead) > that if there are no records then it may due to unknown topic. > But that is another network call to kafka > > > On Fri, Jan 31, 2025 at 3:23 PM Claus Ibsen <claus.ib...@gmail.com> wrote: > >> Hi >> >> >> For Camel in general there is docs on the website, books, videos on >> you-tube and so forth. >> >> This is a volunteer based community, and a bunch of core engineers that >> are very busy. >> I am afraid we don't have excess time to help when its too general and >> environment specific questions/ >> >> There are commercial companies you can contract to get assistance >> https://camel.apache.org/manual/commercial-camel-offerings.html >> >> >> On Fri, Jan 31, 2025 at 2:49 PM Vineet Kumar >> <vineet.ku...@iongroup.com.invalid> wrote: >> >>> Hello Team, >>> >>> I am reaching out on behalf of ION Group regarding some issues we've >>> encountered while working with Apache Camel in our Kafka-based projects. We >>> are seeking your assistance to resolve some challenges related to exception >>> handling in both producer and consumer routes. We have been working with a >>> Kafka producer and consumer route in Camel, and our goal is to catch >>> exceptions if something goes wrong in the Kafka producer or consumer routes. >>> >>> >>> *Project 1: *Latest Version of Libraries (Camel: 4.8.0, Kafka Clients: >>> 3.9.0) - Template POC Project >>> *Findings:* >>> >>> - *Producer Route:* We successfully implemented exception handling >>> for the producer route by adding onException block on global level with >>> maxBlockMs=2500 config. >>> - *This allowed us to catch exceptions within 2.5 seconds if any >>> error occurred during polling, which triggered the OnException block.* >>> - *Consumer Route:* Unfortunately, we were unable to achieve >>> similar results with the consumer route. Even when reproducing an >>> exception >>> (e.g., *UnknownTopicOrPartitionException*) >>> - *The consumer route continued polling indefinitely, and the >>> OnException block was never triggered.* >>> - While we found that we could capture exceptions like * >>> UnknownTopicOrPartitionException *for the producer route, the same >>> did not apply to the consumer route. >>> - There doesn't seem to be a similar config (like maxBlockMs) for >>> the consumer route, and we're unable to handle exceptions in the same >>> way. >>> - We were only able to catch >>> *org.apache.kafka.common.errors.TimeoutException >>> **on which we invoked getCause method to get root cause wrapped >>> exception** i.e, UnknownTopicOrPartitionException* >>> >>> >>> >>> *Project 2: *Older Version of Libraries (Camel: 3.14.10, Kafka Clients: >>> 2.7.2) - *Actual version we want to use* >>> *Findings:* >>> >>> - *Producer Route:* We were able to reproduce a timeout exception, >>> but when calling .getCause() on the Throwable object, we received a >>> null result. >>> - In this version, only the TimeoutException was captured, and the >>> root cause was not available via getCause method invocation. >>> - *Consumer Route:* Similarly, we were unable to catch any >>> exceptions in the consumer route, which resulted in indefinite polling. >>> - We used similar example like producer: We tried consuming from a >>> topic which doesn't exist on the kafka broker >>> - Note:* auto.create.topics.enable=false* was added explicitly in >>> server.properties file used by kafka broker to prevent auto >>> creation of >>> topic >>> >>> >>> >>> *Steps We've Taken:* To resolve these issues, we've tried the following >>> below approaches in both versions and none of them seems to work out for us >>> except onException for producer just for TimeoutException. >>> >>> 1. *OnException: Only caught timeout exceptions in the producer >>> route (Only timeout exceptions)* >>> 2. *ErrorHandler: We tested both custom error and exception handlers >>> without success.* >>> 3. *PollExceptionStrategy* >>> 4. *BreakOnFirstError* >>> 5. *PollOnError* >>> 6. *BridgeErrorHandler* >>> 7. *ExceptionHandler* >>> >>> >>> >>> *Questions:* We would appreciate your guidance on the following: >>> >>> 1. *Are we missing any key configurations or settings that could >>> help resolve this issue to catch any type of exception defined in >>> Errors.class file under kafka-clients library?* >>> 1. *Includes both retriable and non-retriable exceptions* >>> 2. *Could there be an error in our setup or configuration?* >>> 3. *Is there any relevant information missing from the Camel >>> documentation that could help us handle these exceptions properly?* >>> 4. *Is this a known issue or a potential bug in the Camel version >>> we're using in both projects?* >>> 5. *Any other suggestions or insights you may have would be greatly >>> appreciated.* >>> >>> >>> For your reference, I've attached the project zip file on the below >>> Zulip chat link containing relevant information, screenshot including the >>> dependency version details and all necessary project files. >>> Zulip chat link: (46) #general > Assistance Required on Exception >>> Handling in Camel Kafka Con - camel - Zulip >>> <https://camel.zulipchat.com/#narrow/channel/257295-general/topic/Assistance.20Required.20on.20Exception.20Handling.20in.20Camel.20Kafka.20Con> >>> Thank you in advance for your time and support. We look forward to your >>> guidance on this matter. >>> >>> >>> Regards, >>> [image: ION] >>> *Vineet Kumar* >>> vineet.ku...@iongroup.com >>> *m* >>> +91 9003801369 >>> iongroup.com <https://www.iongroup.com/> >>> >>> [image: Impact] >>> <https://www.linkedin.com/showcase/ioncommodities/posts/?feedView=all> >>> >>> >> >> -- >> Claus Ibsen >> ----------------- >> @davsclaus >> Camel in Action 2: https://www.manning.com/ibsen2 >> > > > -- > Claus Ibsen > ----------------- > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > -- Claus Ibsen ----------------- @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
