Hi Thomas, Thanks for your interest in KIP-388. As Ignacio and Radai have mentioned, this <https://github.com/linkedin/kafka/commit/a378c8980af16e3c6d3f6550868ac0fd5a58682e> is our (LinkedIn's) implementation of KIP-388. The implementation and deployment of this broker-side observer has been working very well for us by far. On the other hand, I totally agree with the longer-term concerns raised by other committers. However we still decided to implement the KIP idea as a hot fix in order to solve our immediate problem and meet our business requirements.
The "Rejected Alternatives for Kafka Audit" section at the end of KIP-388 sheds some lights on the client-side auditor/interceptor/observer (sorry about the potential confusion caused by these words being used interchangeably). Best regards, Lincong Li On Wed, Dec 4, 2019 at 8:15 AM Thomas Aley <thomas.a...@ibm.com> wrote: > Thanks for the responses. I did worry about the challenge of exposing a > vast number of internal classes with general interceptor framework. A less > general solution more along the lines of the producer/consumer > interceptors on the client would satisfy the majority of use cases. If we > are smart, we should be able to come up with a pattern that could be > extended further in future if the community sees the demand. > > Looking through the discussion thread for KIP-388, I see a lot of good > points to consider and I intend to dive further into this. > > > Tom Aley > thomas.a...@ibm.com > > > > From: Ismael Juma <ism...@juma.me.uk> > To: Kafka Users <users@kafka.apache.org> > Cc: dev <d...@kafka.apache.org> > Date: 03/12/2019 16:12 > Subject: [EXTERNAL] Re: Broker Interceptors > > > > The main challenge is doing this without exposing a bunch of internal > classes. I haven't seen a proposal that handles that aspect well so far. > > Ismael > > On Tue, Dec 3, 2019 at 7:21 AM Sönke Liebau > <soenke.lie...@opencore.com.invalid> wrote: > > > Hi Thomas, > > > > I think that idea is worth looking at. As you say, if no interceptor is > > configured then the performance overhead should be negligible. Basically > it > > is then up to the user to decide if he wants tomtake the performance > hit. > > We should make sure to think about monitoring capabilities like time > spent > > in the interceptor for records etc. > > > > The most obvious use case I think is server side schema validation, > which > > Confluent are also offering as part of their commercial product, but > other > > ideas come to mind as well. > > > > Best regards, > > Sönke > > > > Thomas Aley <thomas.a...@ibm.com> schrieb am Di., 3. Dez. 2019, 10:45: > > > > > Hi M. Manna, > > > > > > Thank you for your feedback, any and all thoughts on this are > appreciated > > > from the community. > > > > > > I think it is important to distinguish that there are two parts to > this. > > > One would be a server side interceptor framework and the other would > be > > > the interceptor implementations themselves. > > > > > > The idea would be that the Interceptor framework manifests as a plug > > point > > > in the request/response paths that by itself has negligible > performance > > > impact as without an interceptor registered in the framework it is > > > essentially a no-op. This way the out-the-box behavior of the Kafka > > broker > > > remains essentially unchanged, it is only if the cluster administrator > > > registers an interceptor into the framework that the path of a record > is > > > intercepted. This is much like the already accepted and implemented > > client > > > interceptors - the capability exists and it is an opt-in feature. > > > > > > As with the client interceptors and indeed interception in general, > the > > > interceptor implementations need to be thoughtfully crafted to ensure > > > minimal performance impact. Yes the interceptor framework could tap > into > > > nearly everything but would only be tapping into the subset of APIs > that > > > the user wishes to intercept for their use case. > > > > > > Tom Aley > > > thomas.a...@ibm.com > > > > > > > > > > > > From: "M. Manna" <manme...@gmail.com> > > > To: Kafka Users <users@kafka.apache.org> > > > Cc: d...@kafka.apache.org > > > Date: 02/12/2019 11:31 > > > Subject: [EXTERNAL] Re: Broker Interceptors > > > > > > > > > > > > Hi Tom, > > > > > > On Mon, 2 Dec 2019 at 09:41, Thomas Aley <thomas.a...@ibm.com> wrote: > > > > > > > Hi Kafka community, > > > > > > > > I am hoping to get some feedback and thoughts about broker > > interceptors. > > > > > > > > KIP-42 Added Producer and Consumer interceptors which have provided > > > Kafka > > > > users the ability to collect client side metrics and trace the path > of > > > > individual messages end-to-end. > > > > > > > > This KIP also mentioned "Adding message interceptor on the broker > makes > > > a > > > > lot of sense, and will add more detail to monitoring. However, the > > > > proposal is to do it later in a separate KIP". > > > > > > > > One of the motivations for leading with client interceptors was to > gain > > > > experience and see how useable they are before tackling the server > side > > > > implementation which would ultimately "allow us to have a more > > > > complete/detailed message monitoring". > > > > > > > > Broker interceptors could also provide more value than just more > > > complete > > > > and detailed monitoring such as server side schema validation, so I > am > > > > curious to learn if anyone in the community has progressed this > work; > > > has > > > > ideas about other potential server side interceptor uses or has > > actually > > > > implemented something similar. > > > > > > > > > > I personally feel that the cost here is the impact on performance. If > I > > > am > > > right, this interceptor is going to tap into nearly everything. If you > > > have > > > strong guarantee (min.in.sync.replicas = N-1) then this may incur some > > > delay (and let's not forget inter broker comms protection by TLS > config). > > > This may not be desirable for some systems. That said, it would be > good > > to > > > know what others think about this. > > > > > > Thanks, > > > > > > > > > > > Regards, > > > > > > > > Tom Aley > > > > thomas.a...@ibm.com > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > number > > > > 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 > > > 3AU > > > > > > > > > > > > > > > > > > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > number > > > 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > > 3AU > > > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >
