[
https://issues.apache.org/jira/browse/KAFKA-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248887#comment-17248887
]
flavio livide edited comment on KAFKA-6020 at 12/14/20, 10:36 AM:
------------------------------------------------------------------
We have use cases where some consumers need the whole topic and others only a
small subset. The set up is quite dynamic so setting up topics on purpose for
consumers becomes quite complicated to manage.
A very simple form of broker side filter would make a big improvement. Could
be a key prefix, header based, or as brutal as letting a producer set a number
1 to 1024 and use a bitmask to filter for clients (if I can reduce traffic by
three orders of magnitude I can do the rest of the filtering client side).
was (Author: flavio_livide):
We have use cases where some consumers need the whole topic and others only a
small subset. The set up is quite dynamic so setting up topics on purpose for
consumers becomes quite complicated to manage.
A very simple form of broker side filter would make a big improvement. Could be
a key prefix, header based, or as brutal as letting a producer set a number 1
to 1024 and use a bitmask to filter for clients.
> Broker side filtering
> ---------------------
>
> Key: KAFKA-6020
> URL: https://issues.apache.org/jira/browse/KAFKA-6020
> Project: Kafka
> Issue Type: New Feature
> Components: consumer
> Reporter: Pavel Micka
> Priority: Major
> Labels: needs-kip
>
> Currently, it is not possible to filter messages on broker side. Filtering
> messages on broker side is convenient for filter with very low selectivity
> (one message in few thousands). In my case it means to transfer several GB of
> data to consumer, throw it away, take one message and do it again...
> While I understand that filtering by message body is not feasible (for
> performance reasons), I propose to filter just by message key prefix. This
> can be achieved even without any deserialization, as the prefix to be matched
> can be passed as an array (hence the broker would do just array prefix
> compare).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
