[
https://issues.apache.org/jira/browse/KAFKA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7317:
-----------------------------------
Description:
In KAFKA-4633 we switched from "collection subscription" to "pattern
subscription" for `Consumer#subscribe()` to avoid triggering auto topic
creating on the broker. In KAFKA-5291, the metadata request was extended to
overwrite the broker config within the request itself. However, this feature is
only used in `KafkaAdminClient`. KAFKA-7320 adds this feature for the consumer
client, too.
This ticket proposes to use the new feature within Kafka Streams to allow the
usage of collection based subscription in consumer and admit clients to reduce
the metadata response size than can be very large for large number of
partitions in the cluster.
Note, that Streams need to be able to distinguish if it connects to older
brokers that do not support the new metadata request and still use pattern
subscription for this case.
was:
In KAFKA-4633 we switched from "collection subscription" to "pattern
subscription" for `Consumer#subscribe()` to avoid triggering auto topic
creating on the broker. In KAFKA-5291, the metadata request was extended to
overwrite the broker config within the request itself. However, this feature is
only used in `KafkaAdminClient`.
This ticket proposes to use the same feature within Kafka Streams to allow the
usage of collection based subscription in all clients to reduce the metadata
response size than can be very large for large number of partitions in the
cluster.
Also, the new metadata request cannot be used in consumer clients atm. Thus, we
either need an internal config to allow Streams to enable this feature on the
consumer, or we do a KIP and add a public config to enable this feature for all
users.
For the AdminClient that is used during rebalance, Streams should also switch
from wildcard metadata requests to topic-collection requests.
Note, that Streams need to be able to distinguish if it connects to older
brokers that do not support the new metadata request and still use pattern
subscription for this case.
> Use collections subscription for main consumer to reduce metadata
> -----------------------------------------------------------------
>
> Key: KAFKA-7317
> URL: https://issues.apache.org/jira/browse/KAFKA-7317
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
>
> In KAFKA-4633 we switched from "collection subscription" to "pattern
> subscription" for `Consumer#subscribe()` to avoid triggering auto topic
> creating on the broker. In KAFKA-5291, the metadata request was extended to
> overwrite the broker config within the request itself. However, this feature
> is only used in `KafkaAdminClient`. KAFKA-7320 adds this feature for the
> consumer client, too.
> This ticket proposes to use the new feature within Kafka Streams to allow the
> usage of collection based subscription in consumer and admit clients to
> reduce the metadata response size than can be very large for large number of
> partitions in the cluster.
> Note, that Streams need to be able to distinguish if it connects to older
> brokers that do not support the new metadata request and still use pattern
> subscription for this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
