[ https://issues.apache.org/jira/browse/KAFKA-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248887#comment-17248887 ]
flavio livide edited comment on KAFKA-6020 at 12/14/20, 10:36 AM: ------------------------------------------------------------------ We have use cases where some consumers need the whole topic and others only a small subset. The set up is quite dynamic so setting up topics on purpose for consumers becomes quite complicated to manage. A very simple form of broker side filter would make a big improvement. Could be a key prefix, header based, or as brutal as letting a producer set a number 1 to 1024 and use a bitmask to filter for clients (if I can reduce traffic by three orders of magnitude I can do the rest of the filtering client side). was (Author: flavio_livide): We have use cases where some consumers need the whole topic and others only a small subset. The set up is quite dynamic so setting up topics on purpose for consumers becomes quite complicated to manage. A very simple form of broker side filter would make a big improvement. Could be a key prefix, header based, or as brutal as letting a producer set a number 1 to 1024 and use a bitmask to filter for clients. > Broker side filtering > --------------------- > > Key: KAFKA-6020 > URL: https://issues.apache.org/jira/browse/KAFKA-6020 > Project: Kafka > Issue Type: New Feature > Components: consumer > Reporter: Pavel Micka > Priority: Major > Labels: needs-kip > > Currently, it is not possible to filter messages on broker side. Filtering > messages on broker side is convenient for filter with very low selectivity > (one message in few thousands). In my case it means to transfer several GB of > data to consumer, throw it away, take one message and do it again... > While I understand that filtering by message body is not feasible (for > performance reasons), I propose to filter just by message key prefix. This > can be achieved even without any deserialization, as the prefix to be matched > can be passed as an array (hence the broker would do just array prefix > compare). -- This message was sent by Atlassian Jira (v8.3.4#803005)