Hi, thx for reply.we have 2 situation to use server-side filter,that also mean advantage consumer filter doen't have: 1.as Guozhang said,server filter would reduce network cost. we use aws ec2 to provide kafka, all of them and kafka clients in different aws regions to guarantee availability. network flow cross regions means large bill,sometimes even more expensive than several ec2 instances. 2.not only aws, my employer use many public cloud services,and often need use kafka to sync data from one cloud service to another.
now we have some kinds of message,each of them has different tags like messageType,deviceType,manufacturer,city ,and so on.and we have some application,one need to analyze message that messageType=hearbeat & manufacturer=microsoft ,one need analyze message that deviceType=switch & messageType=userOperation. every application consume different tags combination,it's hard to decide send meessage to which topic and recive message from which topic. I know kafka send message use zero-copy, if kafka implement server filter, its code may looks like LazyDownConversionRecords and will cause some performance problem for kafka,but we really think it's necessary. thx. Guozhang Wang <wangg...@gmail.com> 于2019年8月28日周三 下午11:53写道： > The reduction on network cost is a valid point: letting the consumer to do > the filtering would waste the network bandwidth as many bytes would be > dropped on the floor directly afterwards. But note that letting brokers to > filter on fetch response means that we cannot use nio zero-copy as we need > to do this filtering by allocating the memory in user space which could be > a big burden at broker-side. > > Xianyuan: have you considered using different topics instead, i.e. having a > finer-grained grouping of messages based on the consumer subscriptions? > > Guozhang > > On Wed, Aug 28, 2019 at 8:14 AM Adam Bellemare <adam.bellem...@gmail.com> > wrote: > > > I do not think this will be implemented in the server. Why not simply > > create a consumer that filters as required and outputs to a new topic? > > > > > On Aug 28, 2019, at 7:48 AM, Xiangyuan LI <flashmouse2...@gmail.com> > > wrote: > > > > > > Hi, > > > I want to know whether kafka has any plan to filter message on > > > server-side.this can reduce network flow obviously when many groups > > > subscribe the same one topic and only need part of message. > > > > > -- > -- Guozhang >
