The topics of global stores are not included by design. The "problem" is, that each instance needs to consume *all* topic-partitions from and thus topis, we thus they cannot be include into the consumer group that would assign each partition to exactly one instance. Hence, an additional consumer is used that uses partition assignment (instead of subscription) and this consumer does not commit any offset to Kafka.
Note that global stores are bootstrapped before processing begins
though, and are expected to be low throughput topic anyway.
-Matthias
On 11/6/18 2:03 AM, Patrik Kleindl wrote:
> Hello
>
> Am I doing something wrong or is it by design that global state stores and
> their consumers do not show up under the consumer-groups?
> With the consumer group command (and in control center as well) I don't get
> any output for the group:
> ./kafka-consumer-groups --bootstrap-server broker:9092 --group somegroup
> --describe
> Note: This will not show information about old Zookeeper-based consumers.
>
> If I query for the state I get a response that members are present:
> ./kafka-consumer-groups --bootstrap-server broker:9092 --group somegroup
> --describe --state
> Note: This will not show information about old Zookeeper-based consumers.
>
> COORDINATOR (ID) ASSIGNMENT-STRATEGY
> STATE #MEMBERS
> broker:9092 (1) stream Stable 2
>
> This is quite irritating as we cannot see if a global state store has
> caught up with a backlog of messages.
>
> Code to reproduce:
> builder.globalTable(TOPIC_NAME, Materialized
> .<String, String, KeyValueStore<Bytes, byte[]>>as(STORENAME)
> .withKeySerde(Serdes.String())
> .withValueSerde(Serdes.String()));
>
> Nothing fancy.
>
> Logs:
> 2018-11-05 21:25:56 INFO AbstractCoordinator:442 - (Re-)joining group
> 2018-11-05 21:25:56 INFO StreamPartitionAssignor:481 - Assigned tasks to
> clients as {e0250aa5-e1c6-4d33-a746-bc9357c66965=[activeTasks: ([])
> standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([])
> prevAssignedTasks: ([]) capacity: 1]}.
> 2018-11-05 21:25:56 WARN ConsumerCoordinator:376 - The following
> subscribed topics are not assigned to any members: [storetopic]
> 2018-11-05 21:25:56 INFO AbstractCoordinator:409 - Successfully joined
> group with generation 3
> 2018-11-05 21:25:56 INFO ConsumerCoordinator:256 - Setting newly assigned
> partitions []
>
> The store works after this, but it is not shown.
>
> Any input is appreciated
>
> best regards
>
> Patrik
>
> PS: The customer will forward this to the Confluent support too, but I'm
> asking here for public visibility
>
signature.asc
Description: OpenPGP digital signature
