Greetings all,
Please correct me if this is not the right place to discuss this. I filed
a jira about this but I figured the mailing list might encourage better
collaboration.
I'd like to propose adding the ProducerRecord field to the
onAcknowledgement callback in the ProducerInterceptor interface that was
added in KAFKA-3162
The issue I'm having is that its not possible to determine if a specific
record was sent (based on the record content or key), only that some record
was sent based on the RecordMetadata. From reading the KIP and the initial
mailing list discussion, it seems that the design was modeled after the
existing Producer.Callback interface. This model makes sense given that
callbacks are typically anonymous functions where you would have access to
the producer record:
ProducerRecord record = ...;
producer.send(record, new Callback() {
@Override
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
// do something with record
}
})
However, based on my understanding of the interceptor design, the
interceptor should be thread safe and thus stateless. Even if you wanted to
make it stateful, its not easy to tie together a record from onSend to
onAcknowledgement. Exception handling also suffers from this if you want
to tie the exception to the specific record that failed
With that in mind, is there a way that the producer interceptor API can be
modified so that implementers can be aware of the key & content of the
record that was acknowledged or failed?
Thanks,
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
