[ https://issues.apache.org/jira/browse/KAFKA-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456955#comment-17456955 ]
Matthias J. Sax edited comment on KAFKA-13500 at 12/10/21, 8:32 AM: -------------------------------------------------------------------- Not 100% sure: for individual configs, maybe. – But it won't solve the monitoring issue. Users might still see that the "restore consumer" has a non-zero lag if they monitor the consumer lag metric if there are lagging standbys and we are not actually restoring state for active tasks. I marked this ticket as "minor", so if we think it's not worth it (for now?) for with me. But I don't think that our code would become (much) more complicated and thus, it might be an easy win... (the main impact might be that we would need to add a new `standby.producer` config prefix... so we might technically need a KIP) Personally still in favor to add a new consumer, but I leave it up to you. Also, we can always make this change later on, too. was (Author: mjsax): Not 100% sure: for individual configs, maybe. – But it won't solve the monitoring issue. Users might still see that the "restore consumer" has a non-zero lag if they monitor the consumer lag metric if there a lagging standbys and we are not actually restoring state for active tasks. I marked this ticket as "minor", so if we think it's not worth it (for now?) for with me. But I don't think that our code would become (much) more complicated and thus, it might be an easy win... (the main impact might be that we would need to add a new `standby.producer` config prefix... so we might technically need a KIP) Personally still in favor to add a new consumer, but I leave it up to you. Also, we can always make this change later on, too. > Consider adding a dedicated standby consumer > -------------------------------------------- > > Key: KAFKA-13500 > URL: https://issues.apache.org/jira/browse/KAFKA-13500 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: Matthias J. Sax > Priority: Minor > > We currently use the restore consumer to recover state for active tasks and > to maintain standby tasks during regular processing. This setup has a few > disadvantages > # During state recovery, we might want to apply different consumer configs > compared to standby maintenance during regular processing. > # It make monitoring confusing: because we never commit offsets for > changelog topics, users can only monitor the client's "lag metric" to > observer restore progress (without the need to register a restore listener). > However, if they are interesting in a restore metric, during regular > processing it would report the standby lag, which can be rather confusing. > Because the restore consumer does not use consumer group management, it seems > to be low overhead to actually use a third consumer, because there won't be > any heartbeat thread. -- This message was sent by Atlassian Jira (v8.20.1#820001)