Hello Team,
I am reaching out on behalf of ION Group regarding some issues we've
encountered while working with Apache Camel in our Kafka-based projects. We are
seeking your assistance to resolve some challenges related to exception
handling in both producer and consumer routes. We have been working with a
Kafka producer and consumer route in Camel, and our goal is to catch exceptions
if something goes wrong in the Kafka producer or consumer routes.
Project 1: Latest Version of Libraries (Camel: 4.8.0, Kafka Clients: 3.9.0) -
Template POC Project
Findings:
*
Producer Route: We successfully implemented exception handling for the producer
route by adding onException block on global level with maxBlockMs=2500 config.
*
This allowed us to catch exceptions within 2.5 seconds if any error occurred
during polling, which triggered the OnException block.
*
Consumer Route: Unfortunately, we were unable to achieve similar results with
the consumer route. Even when reproducing an exception (e.g.,
UnknownTopicOrPartitionException)
*
The consumer route continued polling indefinitely, and the OnException block
was never triggered.
*
While we found that we could capture exceptions like
UnknownTopicOrPartitionException for the producer route, the same did not apply
to the consumer route.
*
There doesn't seem to be a similar config (like maxBlockMs) for the consumer
route, and we're unable to handle exceptions in the same way.
*
We were only able to catch org.apache.kafka.common.errors.TimeoutException on
which we invoked getCause method to get root cause wrapped exception i.e,
UnknownTopicOrPartitionException
Project 2: Older Version of Libraries (Camel: 3.14.10, Kafka Clients: 2.7.2) -
Actual version we want to use
Findings:
*
Producer Route: We were able to reproduce a timeout exception, but when calling
.getCause() on the Throwable object, we received a null result.
*
In this version, only the TimeoutException was captured, and the root cause was
not available via getCause method invocation.
*
Consumer Route: Similarly, we were unable to catch any exceptions in the
consumer route, which resulted in indefinite polling.
*
We used similar example like producer: We tried consuming from a topic which
doesn't exist on the kafka broker
*
Note: auto.create.topics.enable=false was added explicitly in server.properties
file used by kafka broker to prevent auto creation of topic
Steps We've Taken: To resolve these issues, we've tried the following below
approaches in both versions and none of them seems to work out for us except
onException for producer just for TimeoutException.
1.
OnException: Only caught timeout exceptions in the producer route (Only timeout
exceptions)
2.
ErrorHandler: We tested both custom error and exception handlers without
success.
3. PollExceptionStrategy
4. BreakOnFirstError
5. PollOnError
6. BridgeErrorHandler
7.
ExceptionHandler
Questions: We would appreciate your guidance on the following:
1.
Are we missing any key configurations or settings that could help resolve this
issue to catch any type of exception defined in Errors.class file under
kafka-clients library?
*
Includes both retriable and non-retriable exceptions
2. Could there be an error in our setup or configuration?
3. Is there any relevant information missing from the Camel documentation
that could help us handle these exceptions properly?
4. Is this a known issue or a potential bug in the Camel version we're using
in both projects?
5.
Any other suggestions or insights you may have would be greatly appreciated.
For your reference, I've attached the project zip file on the below Zulip chat
link containing relevant information, screenshot including the dependency
version details and all necessary project files.
Zulip chat link: (46) #general > Assistance Required on Exception Handling in
Camel Kafka Con - camel -
Zulip<https://camel.zulipchat.com/#narrow/channel/257295-general/topic/Assistance.20Required.20on.20Exception.20Handling.20in.20Camel.20Kafka.20Con>
Thank you in advance for your time and support. We look forward to your
guidance on this matter.
Regards,
[ION]
Vineet Kumar
[email protected]<mailto:[email protected]>
m
+91 9003801369
iongroup.com<https://www.iongroup.com/>
[Impact]<https://www.linkedin.com/showcase/ioncommodities/posts/?feedView=all>
