Hi Joel, Yes, we considered interceptors on broker side -- they make a lot of sense and will add more detail to monitoring. We propose to do it later in a separate KIP because: 1) broker interceptors are more risky, since brokers are more sensitive to overheads; 2) Just producer and consumer interceptors will enable end-to-end monitoring and thus a better reward vs. risk tradeoff; 3) Once we have producer and consumer interceptors, we gain more experience and see usability; 4) Once we see usability, we can start working on broker interceptors as separate KIP.
I added broker interceptors to Rejected Alternatives section (out-of-scope for this KIP). Let me know your thoughts. Thanks, Anna On Mon, Jan 25, 2016 at 7:37 PM, Jay Kreps <j...@confluent.io> wrote: > I think there is some tension here between exposing lots of hooks that you > can implement and giving a simple, supportable interface. > > Personally I would actually argue for not adding an onDequeued but actually > removing both the onEnqueued() and onReceived(). > > My argument is that onSend and onAcknowledgment are fundamental aspects of > the producer protocol. This means they are easy to understand, and they > don't depend on our implementation at all. I think this implies we can > commit to the API over the long haul. The same is true for the consumer for > onConsume and onCommit. > > For example whether onConsume should take the serialized or deserialized > value is very dependent on whether we choose to do serialization lazily or > not which is still totally a matter of debate. > > I think maybe the best way to figure this out would be to try to come up > with a set of concrete use cases that justify each method. And then we can > think if it is worth it to include or not. I am a little wary of arguments > along the lines of "it could be useful" because of course you could say > that about any point in the code path. > > For onDequeued, I think we already have the queue time metric in the > producer, right? I think that provides insight into the time spent in the > queue without a need for a plugin, right? (I may have misunderstood the use > case, though). Are there other use cases? > > -Jay > > > On Mon, Jan 25, 2016 at 12:02 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Nice KIP. Excellent idea. > > Was just thinking if we can add onDequed() to the ProducerIterceptor > > interface. Since we have the onEnqueued(), it will help the client or the > > tools to know how much time the message spent in the RecordAccumulator. > > Also an API to check if there are any messages left for a particular > topic > > in the RecordAccumulator would help. > > > > Thanks, > > > > Mayuresh > > > > On Mon, Jan 25, 2016 at 11:29 AM, Todd Palino <tpal...@gmail.com> wrote: > > > > > Great idea. I’ve been talking about this for 2 years, and I’m glad > > someone > > > is finally picking it up. Will take a look at the KIP at some point > > > shortly. > > > > > > -Todd > > > > > > > > > On Mon, Jan 25, 2016 at 11:24 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > Hey Becket, > > > > > > > > Yeah this is really similar to the callback. The difference is really > > in > > > > who sets the behavior. The idea of the interceptor is that it doesn't > > > > require any code change in apps so you can globally add behavior to > > your > > > > Kafka usage without changing app code. Whereas the callback is added > by > > > the > > > > app. The idea is to kind of obviate the need for the wrapper code > that > > > e.g. > > > > LinkedIn maintains to hold this kind of stuff. > > > > > > > > -Jay > > > > > > > > On Sun, Jan 24, 2016 at 4:21 PM, Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > This could be a useful feature. And I think there are some use > cases > > to > > > > > mutate the data like rejected alternative one mentioned. > > > > > > > > > > I am wondering if there is functional overlapping between > > > > > ProducerInterceptor.onAcknowledgement() and the producer callback? > I > > > can > > > > > see that the Callback could be a per record setting while > > > > > onAcknowledgement() is a producer level setting. Other than that, > is > > > > there > > > > > any difference between them? > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On Fri, Jan 22, 2016 at 6:21 PM, Neha Narkhede <n...@confluent.io> > > > > wrote: > > > > > > > > > > > James, > > > > > > > > > > > > That is one of the many monitoring use cases for the interceptor > > > > > interface. > > > > > > > > > > > > Thanks, > > > > > > Neha > > > > > > > > > > > > On Fri, Jan 22, 2016 at 6:18 PM, James Cheng <jch...@tivo.com> > > > wrote: > > > > > > > > > > > > > Anna, > > > > > > > > > > > > > > I'm trying to understand a concrete use case. It sounds like > > > producer > > > > > > > interceptors could be used to implement part of LinkedIn's > Kafak > > > > Audit > > > > > > > tool? > https://engineering.linkedin.com/kafka/running-kafka-scale > > > > > > > > > > > > > > Part of that is done by a wrapper library around the kafka > > producer > > > > > that > > > > > > > keeps a count of the number of messages produced, and then > sends > > > that > > > > > > count > > > > > > > to a side-topic. It sounds like the producer interceptors could > > > > > possibly > > > > > > be > > > > > > > used to implement that? > > > > > > > > > > > > > > -James > > > > > > > > > > > > > > > On Jan 22, 2016, at 4:33 PM, Anna Povzner <a...@confluent.io > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I just created a KIP-42 for adding producer and consumer > > > > interceptors > > > > > > for > > > > > > > > intercepting messages at different points on producer and > > > consumer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors > > > > > > > > > > > > > > > > Comments and suggestions are welcome! > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Anna > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > This email and any attachments may contain confidential and > > > > privileged > > > > > > > material for the sole use of the intended recipient. Any > review, > > > > > copying, > > > > > > > or distribution of this email (or any attachments) by others is > > > > > > prohibited. > > > > > > > If you are not the intended recipient, please contact the > sender > > > > > > > immediately and permanently delete this email and any > > attachments. > > > No > > > > > > > employee or agent of TiVo Inc. is authorized to conclude any > > > binding > > > > > > > agreement on behalf of TiVo Inc. by email. Binding agreements > > with > > > > TiVo > > > > > > > Inc. may only be made by a signed written agreement. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Thanks, > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > *—-* > > > *Todd Palino* > > > Staff Site Reliability Engineer > > > Data Infrastructure Streaming > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > >
