[
https://issues.apache.org/jira/browse/KAFKA-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876309#comment-16876309
]
Arjun Satish edited comment on KAFKA-8597 at 7/1/19 4:10 PM:
-------------------------------------------------------------
Thanks for the feature request, [~conker84].
I think there might be more value in integrating the reporter framework [1], so
we can tap into existing reporters (currently, we have two, including the DLQ).
That way, if a DLQ is disabled by the end-user, calls to report(...) will
respect that. But as Randall mentioned, introducing additional APIs comes with
a catch.
An alternative could be to raise a new Exception class (say, BadRecordException
that extends RetriableException) that can inform the framework of bad records.
Here, we might only have to think about how to serialize records into the DLQ
(the framework might not be holding on to these original records if the
connector buffers records).
[1]
[https://github.com/apache/kafka/blob/2.3.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java#L245-L247
was (Author: wicknicks):
Thanks for the feature request, [~conker84].
I think there might be more value in integrating the reporter framework [1], so
we can tap into existing reporters (currently, we have two, including the DLQ).
That way, if a DLQ is disabled by the end-user, calls to report(...) will
respect that. But as Randall mentioned, introducing additional APIs comes with
a catch.
An alternative could be to raise a new Exception class (say, BadRecordException
that extends RetriableException) that can inform the framework of bad records.
Here, we might only have to think about how to serialize records into the DLQ
(the framework might not be holding on to these original records if the
connector buffers records).
[1]
[https://github.com/apache/kafka/blob/2.3.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java#L245-L247
]
> Give access to the Dead Letter Queue APIs to Kafka Connect Developers
> ---------------------------------------------------------------------
>
> Key: KAFKA-8597
> URL: https://issues.apache.org/jira/browse/KAFKA-8597
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Andrea Santurbano
> Priority: Major
> Fix For: 2.4.0
>
>
> Would be cool to have the chance to have access to the DLQ APIs in order to
> enable us (developers) to use that.
> For instance, if someone uses JSON as message format with no schema and it's
> trying to import some data into a table, and the JSON contains a null value
> for a NON-NULL table field, so we want to move that event to the DLQ.
> Thanks a lot!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
