Yash Mayya created KAFKA-14455:
----------------------------------
Summary: Kafka Connect create and update REST APIs should surface
failures while writing to the config topic
Key: KAFKA-14455
URL: https://issues.apache.org/jira/browse/KAFKA-14455
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Yash Mayya
Assignee: Yash Mayya
Kafka Connect's `POST /connectors` and `PUT /connectors/\{connector}/config`
REST APIs internally simply write a message to the Connect cluster's internal
config topic (which is then processed asynchronously by the herder). However,
no callback is passed to the producer's send method and there is no error
handling in place for producer send failures (see
[here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716]
/
[here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726]).
Consider one such case where the Connect worker's principal doesn't have a
WRITE ACL on the cluster's config topic. Now suppose the user submits a
connector's configs via one of the above two APIs. The producer send
[here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716]
/
[here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726]
won't succeed (due to a TopicAuthorizationException) but the API responses
will be `201 Created` success responses anyway. This is a very poor UX because
the connector will actually never be created but the API response indicated
success. Furthermore, this failure would only be detectable if TRACE logs are
enabled (via [this
log)|https://github.com/apache/kafka/blob/df29b17fc40f7c15460988d58bc652c3d66b60f8/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java]
making it near impossible for users to debug. Producer callbacks should be
used to surface write failures back to the user via the API response.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
