I think there is some tension here between exposing lots of hooks that you can implement and giving a simple, supportable interface.
Personally I would actually argue for not adding an onDequeued but actually removing both the onEnqueued() and onReceived(). My argument is that onSend and onAcknowledgment are fundamental aspects of the producer protocol. This means they are easy to understand, and they don't depend on our implementation at all. I think this implies we can commit to the API over the long haul. The same is true for the consumer for onConsume and onCommit. For example whether onConsume should take the serialized or deserialized value is very dependent on whether we choose to do serialization lazily or not which is still totally a matter of debate. I think maybe the best way to figure this out would be to try to come up with a set of concrete use cases that justify each method. And then we can think if it is worth it to include or not. I am a little wary of arguments along the lines of "it could be useful" because of course you could say that about any point in the code path. For onDequeued, I think we already have the queue time metric in the producer, right? I think that provides insight into the time spent in the queue without a need for a plugin, right? (I may have misunderstood the use case, though). Are there other use cases? -Jay On Mon, Jan 25, 2016 at 12:02 PM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Nice KIP. Excellent idea. > Was just thinking if we can add onDequed() to the ProducerIterceptor > interface. Since we have the onEnqueued(), it will help the client or the > tools to know how much time the message spent in the RecordAccumulator. > Also an API to check if there are any messages left for a particular topic > in the RecordAccumulator would help. > > Thanks, > > Mayuresh > > On Mon, Jan 25, 2016 at 11:29 AM, Todd Palino <tpal...@gmail.com> wrote: > > > Great idea. I’ve been talking about this for 2 years, and I’m glad > someone > > is finally picking it up. Will take a look at the KIP at some point > > shortly. > > > > -Todd > > > > > > On Mon, Jan 25, 2016 at 11:24 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > Hey Becket, > > > > > > Yeah this is really similar to the callback. The difference is really > in > > > who sets the behavior. The idea of the interceptor is that it doesn't > > > require any code change in apps so you can globally add behavior to > your > > > Kafka usage without changing app code. Whereas the callback is added by > > the > > > app. The idea is to kind of obviate the need for the wrapper code that > > e.g. > > > LinkedIn maintains to hold this kind of stuff. > > > > > > -Jay > > > > > > On Sun, Jan 24, 2016 at 4:21 PM, Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > This could be a useful feature. And I think there are some use cases > to > > > > mutate the data like rejected alternative one mentioned. > > > > > > > > I am wondering if there is functional overlapping between > > > > ProducerInterceptor.onAcknowledgement() and the producer callback? I > > can > > > > see that the Callback could be a per record setting while > > > > onAcknowledgement() is a producer level setting. Other than that, is > > > there > > > > any difference between them? > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On Fri, Jan 22, 2016 at 6:21 PM, Neha Narkhede <n...@confluent.io> > > > wrote: > > > > > > > > > James, > > > > > > > > > > That is one of the many monitoring use cases for the interceptor > > > > interface. > > > > > > > > > > Thanks, > > > > > Neha > > > > > > > > > > On Fri, Jan 22, 2016 at 6:18 PM, James Cheng <jch...@tivo.com> > > wrote: > > > > > > > > > > > Anna, > > > > > > > > > > > > I'm trying to understand a concrete use case. It sounds like > > producer > > > > > > interceptors could be used to implement part of LinkedIn's Kafak > > > Audit > > > > > > tool? https://engineering.linkedin.com/kafka/running-kafka-scale > > > > > > > > > > > > Part of that is done by a wrapper library around the kafka > producer > > > > that > > > > > > keeps a count of the number of messages produced, and then sends > > that > > > > > count > > > > > > to a side-topic. It sounds like the producer interceptors could > > > > possibly > > > > > be > > > > > > used to implement that? > > > > > > > > > > > > -James > > > > > > > > > > > > > On Jan 22, 2016, at 4:33 PM, Anna Povzner <a...@confluent.io> > > > wrote: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I just created a KIP-42 for adding producer and consumer > > > interceptors > > > > > for > > > > > > > intercepting messages at different points on producer and > > consumer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors > > > > > > > > > > > > > > Comments and suggestions are welcome! > > > > > > > > > > > > > > Thanks, > > > > > > > Anna > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > This email and any attachments may contain confidential and > > > privileged > > > > > > material for the sole use of the intended recipient. Any review, > > > > copying, > > > > > > or distribution of this email (or any attachments) by others is > > > > > prohibited. > > > > > > If you are not the intended recipient, please contact the sender > > > > > > immediately and permanently delete this email and any > attachments. > > No > > > > > > employee or agent of TiVo Inc. is authorized to conclude any > > binding > > > > > > agreement on behalf of TiVo Inc. by email. Binding agreements > with > > > TiVo > > > > > > Inc. may only be made by a signed written agreement. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Neha > > > > > > > > > > > > > > > > > > > > -- > > *—-* > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
