Re: [Discuss] Question on KIP-298: Error Handling in Kafka Connect

2019-02-06 Thread Randall Hauch
Hi, Pere.

The primary reason that KIP-298 did not support a DLQ for source connectors
was because we couldn't get around serialization problems. With source
connectors, the converter (serializer) is the last element in the chain,
and if there is a problem serializing a record then we could not work out
how to serialize the unserializeable record so it could be written to the
DLQ? Another problem was the inability to write into Kafka, at which point
it's probably likely we cannot write to the DLQ topic, either.

Randall

On Tue, Jan 1, 2019 at 2:18 PM Pere Urbón Bayes 
wrote:

> Hi,
>  a quick question on the KIP-298 Dead letter queue, as I read from the KIP
> is only available for the Sink connectors.
>
> While I know the challenges of defining a dead-letter queue for the
> incoming messages, I wanted ask/discuss what is the sense in here for this,
> do you completely discard the option?
>
> I sort of see it useful for messages that where pulled from the source, but
> somehow could not be ingested in Kafka, might be because of serialisation
> for example.
>
> What do you think?
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>


[Discuss] Question on KIP-298: Error Handling in Kafka Connect

2019-01-01 Thread Pere Urbón Bayes
Hi,
 a quick question on the KIP-298 Dead letter queue, as I read from the KIP
is only available for the Sink connectors.

While I know the challenges of defining a dead-letter queue for the
incoming messages, I wanted ask/discuss what is the sense in here for this,
do you completely discard the option?

I sort of see it useful for messages that where pulled from the source, but
somehow could not be ingested in Kafka, might be because of serialisation
for example.

What do you think?

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/