Jason Gustafson created KAFKA-12361:
---------------------------------------
Summary: Change default connect producer request timeout
Key: KAFKA-12361
URL: https://issues.apache.org/jira/browse/KAFKA-12361
Project: Kafka
Issue Type: Improvement
Reporter: Jason Gustafson
Prior to KIP-91, which introduced delivery.timeout.ms, there was no easy way to
ensure that records sent through the producer would even have the opportunity
to get delivered. Records could be timed out in the accumulator if
`request.timeout.ms` was reached before getting sent. Users worked around this
problem by setting `request.timeout.ms=Int.MaxValue`. The downside is that this
made the producer slower to discover "unclean" connection failures. Now that we
have KIP-91, there shouldn't be any reason to keep this workaround.
One place it would be good to fix this is in connect's source tasks:
{code}
// These settings will execute infinite retries on retriable
exceptions. They *may* be overridden via configs passed to the worker,
// but this may compromise the delivery guarantees of Kafka Connect.
producerProps.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,
Integer.toString(Integer.MAX_VALUE));
{code}
The comment about delivery guarantees is a little vague, but I think mainly it
is what was discussed above about ensuring at least once delivery. Note that
none of the default configs including request timeout or delivery timeout can
avoid duplicates in all cases. For that idempotence is needed. It is worth
considering separately for connect whether that should be the default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
