At LinkedIn we run a style of "read-only" interceptor we call an observer. We use this for usage monitoring. https://github.com/linkedin/kafka/commit/a378c8980af16e3c6d3f6550868ac0fd5a58682e
There is always a tension between exposing internals, creating stable interfaces and performance. It's understandable that upstream feels uneasy about this type of change. For us, the observer has become an essential tool, so we're ok if we need to update code. Personally, I would like to argue that a general interceptor framework would have been valuable from the very beginning. Since one didn't exist, we've had to find one-off solutions for a few different problems. A few examples that may have fit in there include up/down convert formats, authorizer, quotas, transaction coordinator, idempotent producer, etc. Ok, ok, I'm overreaching, but you get the idea. The chain of processes that a message/request goes through are basically interceptors, and we have decided that instead of a generic framework, we prefer to do the one-offs. Nacho On Tue, Dec 3, 2019 at 8:03 AM Ismael Juma <ism...@juma.me.uk> wrote: > The main challenge is doing this without exposing a bunch of internal > classes. I haven't seen a proposal that handles that aspect well so far. > > Ismael > > On Tue, Dec 3, 2019 at 7:21 AM Sönke Liebau > <soenke.lie...@opencore.com.invalid> wrote: > > > Hi Thomas, > > > > I think that idea is worth looking at. As you say, if no interceptor is > > configured then the performance overhead should be negligible. Basically > it > > is then up to the user to decide if he wants tomtake the performance hit. > > We should make sure to think about monitoring capabilities like time > spent > > in the interceptor for records etc. > > > > The most obvious use case I think is server side schema validation, which > > Confluent are also offering as part of their commercial product, but > other > > ideas come to mind as well. > > > > Best regards, > > Sönke > > > > Thomas Aley <thomas.a...@ibm.com> schrieb am Di., 3. Dez. 2019, 10:45: > > > > > Hi M. Manna, > > > > > > Thank you for your feedback, any and all thoughts on this are > appreciated > > > from the community. > > > > > > I think it is important to distinguish that there are two parts to > this. > > > One would be a server side interceptor framework and the other would be > > > the interceptor implementations themselves. > > > > > > The idea would be that the Interceptor framework manifests as a plug > > point > > > in the request/response paths that by itself has negligible performance > > > impact as without an interceptor registered in the framework it is > > > essentially a no-op. This way the out-the-box behavior of the Kafka > > broker > > > remains essentially unchanged, it is only if the cluster administrator > > > registers an interceptor into the framework that the path of a record > is > > > intercepted. This is much like the already accepted and implemented > > client > > > interceptors - the capability exists and it is an opt-in feature. > > > > > > As with the client interceptors and indeed interception in general, the > > > interceptor implementations need to be thoughtfully crafted to ensure > > > minimal performance impact. Yes the interceptor framework could tap > into > > > nearly everything but would only be tapping into the subset of APIs > that > > > the user wishes to intercept for their use case. > > > > > > Tom Aley > > > thomas.a...@ibm.com > > > > > > > > > > > > From: "M. Manna" <manme...@gmail.com> > > > To: Kafka Users <us...@kafka.apache.org> > > > Cc: dev@kafka.apache.org > > > Date: 02/12/2019 11:31 > > > Subject: [EXTERNAL] Re: Broker Interceptors > > > > > > > > > > > > Hi Tom, > > > > > > On Mon, 2 Dec 2019 at 09:41, Thomas Aley <thomas.a...@ibm.com> wrote: > > > > > > > Hi Kafka community, > > > > > > > > I am hoping to get some feedback and thoughts about broker > > interceptors. > > > > > > > > KIP-42 Added Producer and Consumer interceptors which have provided > > > Kafka > > > > users the ability to collect client side metrics and trace the path > of > > > > individual messages end-to-end. > > > > > > > > This KIP also mentioned "Adding message interceptor on the broker > makes > > > a > > > > lot of sense, and will add more detail to monitoring. However, the > > > > proposal is to do it later in a separate KIP". > > > > > > > > One of the motivations for leading with client interceptors was to > gain > > > > experience and see how useable they are before tackling the server > side > > > > implementation which would ultimately "allow us to have a more > > > > complete/detailed message monitoring". > > > > > > > > Broker interceptors could also provide more value than just more > > > complete > > > > and detailed monitoring such as server side schema validation, so I > am > > > > curious to learn if anyone in the community has progressed this work; > > > has > > > > ideas about other potential server side interceptor uses or has > > actually > > > > implemented something similar. > > > > > > > > > > I personally feel that the cost here is the impact on performance. If > I > > > am > > > right, this interceptor is going to tap into nearly everything. If you > > > have > > > strong guarantee (min.in.sync.replicas = N-1) then this may incur some > > > delay (and let's not forget inter broker comms protection by TLS > config). > > > This may not be desirable for some systems. That said, it would be good > > to > > > know what others think about this. > > > > > > Thanks, > > > > > > > > > > > Regards, > > > > > > > > Tom Aley > > > > thomas.a...@ibm.com > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > number > > > > 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 > > > 3AU > > > > > > > > > > > > > > > > > > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > number > > > 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > > 3AU > > > > > > > > > -- Nacho - Ignacio Solis - iso...@igso.net
