[jira] [Commented] (KAFKA-16264) Expose `producer.id.expiration.check.interval.ms` as dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844721#comment-17844721 ] Justine Olshan commented on KAFKA-16264: Thanks. I will take a look. It just occurred to me that such a change may require a small KIP. [https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals] Let me double check and get back to you. > Expose `producer.id.expiration.check.interval.ms` as dynamic broker > configuration > - > > Key: KAFKA-16264 > URL: https://issues.apache.org/jira/browse/KAFKA-16264 > Project: Kafka > Issue Type: Improvement >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Major > > Dealing with a scenario where too many producer ids lead to issues (e.g. high > cpu utilization, see KAFKA-16229) put operators in need to flush producer ids > more promptly than usual. > Currently, only the expiration timeout `producer.id.expiration.ms` is exposed > as dynamic config. This is helpful (e.g. by reducing the timeout, less > producer would eventually be kept in memory), but not enough if the > evaluation frequency is not sufficiently short to flush producer ids before > becoming an issue. Only by tuning both, the issue could be workaround. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16264) Expose `producer.id.expiration.check.interval.ms` as dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844484#comment-17844484 ] Jorge Esteban Quilcate Otoya commented on KAFKA-16264: -- Thanks [~jolshan] ! Finally got some time to look into this one. Please, have a look at [https://github.com/apache/kafka/pull/15888] > Expose `producer.id.expiration.check.interval.ms` as dynamic broker > configuration > - > > Key: KAFKA-16264 > URL: https://issues.apache.org/jira/browse/KAFKA-16264 > Project: Kafka > Issue Type: Improvement >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Major > > Dealing with a scenario where too many producer ids lead to issues (e.g. high > cpu utilization, see KAFKA-16229) put operators in need to flush producer ids > more promptly than usual. > Currently, only the expiration timeout `producer.id.expiration.ms` is exposed > as dynamic config. This is helpful (e.g. by reducing the timeout, less > producer would eventually be kept in memory), but not enough if the > evaluation frequency is not sufficiently short to flush producer ids before > becoming an issue. Only by tuning both, the issue could be workaround. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16264) Expose `producer.id.expiration.check.interval.ms` as dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818064#comment-17818064 ] Justine Olshan commented on KAFKA-16264: Hey [~jeqo] thanks for filing. I was also thinking about this after you filed the first ticket. One thing that is interesting is right now the expiration is scheduled on a separate thread on startup. I guess the best course of action is to cancel that scheduled task and create a new one? See UnifiedLog producerExpireCheck. > Expose `producer.id.expiration.check.interval.ms` as dynamic broker > configuration > - > > Key: KAFKA-16264 > URL: https://issues.apache.org/jira/browse/KAFKA-16264 > Project: Kafka > Issue Type: Improvement >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Major > > Dealing with a scenario where too many producer ids lead to issues (e.g. high > cpu utilization, see KAFKA-16229) put operators in need to flush producer ids > more promptly than usual. > Currently, only the expiration timeout `producer.id.expiration.ms` is exposed > as dynamic config. This is helpful (e.g. by reducing the timeout, less > producer would eventually be kept in memory), but not enough if the > evaluation frequency is not sufficiently short to flush producer ids before > becoming an issue. Only by tuning both, the issue could be workaround. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16264) Expose `producer.id.expiration.check.interval.ms` as dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817867#comment-17817867 ] Jorge Esteban Quilcate Otoya commented on KAFKA-16264: -- cc [~jolshan] – as related to https://issues.apache.org/jira/browse/KAFKA-16229 > Expose `producer.id.expiration.check.interval.ms` as dynamic broker > configuration > - > > Key: KAFKA-16264 > URL: https://issues.apache.org/jira/browse/KAFKA-16264 > Project: Kafka > Issue Type: Improvement >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Major > > Dealing with a scenario where too many producer ids lead to issues (e.g. high > cpu utilization, see KAFKA-16229) put operators in need to flush producer ids > more promptly than usual. > Currently, only the expiration timeout `producer.id.expiration.ms` is exposed > as dynamic config. This is helpful (e.g. by reducing the timeout, less > producer would eventually be kept in memory), but not enough if the > evaluation frequency is not sufficiently short to flush producer ids before > becoming an issue. Only by tuning both, the issue could be workaround. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
