[jira] [Updated] (KAFKA-14224) Consumer should auto-commit prior to cooperative partition revocation
[ https://issues.apache.org/jira/browse/KAFKA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-14224: Description: With the old "eager" reassignment logic, we always revoked all partitions prior to each rebalance. When auto-commit is enabled, a part of this process is committing current positions. Under the new "cooperative" logic, we defer revocation until after the rebalance, which means we can continue fetching while the rebalance is in progress. However, when reviewing KAFKA-14196, we noticed that there is no similar logic to commit offsets prior to this deferred revocation. This means that cooperative consumption is more likely to lead to have duplicate consumption even when there is no failure involved. (was: With the old "eager" reassignment logic, we always revoked all partitions prior to each rebalance. When auto-commit is enabled, a part of this process is committing current position. Under the new "cooperative" logic, we defer revocation until after the rebalance, which means we can continue fetching while the rebalance is in progress. However, when reviewing KAFKA-14196, we noticed that there is no similar logic to commit offsets prior to this deferred revocation. This means that cooperative consumption is more likely to lead to have duplicate consumption even when there is no failure involved.) > Consumer should auto-commit prior to cooperative partition revocation > - > > Key: KAFKA-14224 > URL: https://issues.apache.org/jira/browse/KAFKA-14224 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Major > Labels: new-consumer-threading-should-fix > > With the old "eager" reassignment logic, we always revoked all partitions > prior to each rebalance. When auto-commit is enabled, a part of this process > is committing current positions. Under the new "cooperative" logic, we defer > revocation until after the rebalance, which means we can continue fetching > while the rebalance is in progress. However, when reviewing KAFKA-14196, we > noticed that there is no similar logic to commit offsets prior to this > deferred revocation. This means that cooperative consumption is more likely > to lead to have duplicate consumption even when there is no failure involved. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14224) Consumer should auto-commit prior to cooperative partition revocation
[ https://issues.apache.org/jira/browse/KAFKA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-14224: -- Labels: new-consumer-threading-should-fix (was: ) > Consumer should auto-commit prior to cooperative partition revocation > - > > Key: KAFKA-14224 > URL: https://issues.apache.org/jira/browse/KAFKA-14224 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Major > Labels: new-consumer-threading-should-fix > > With the old "eager" reassignment logic, we always revoked all partitions > prior to each rebalance. When auto-commit is enabled, a part of this process > is committing current position. Under the new "cooperative" logic, we defer > revocation until after the rebalance, which means we can continue fetching > while the rebalance is in progress. However, when reviewing KAFKA-14196, we > noticed that there is no similar logic to commit offsets prior to this > deferred revocation. This means that cooperative consumption is more likely > to lead to have duplicate consumption even when there is no failure involved. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14224) Consumer should auto-commit prior to cooperative partition revocation
[ https://issues.apache.org/jira/browse/KAFKA-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-14224: Description: With the old "eager" reassignment logic, we always revoked all partitions prior to each rebalance. When auto-commit is enabled, a part of this process is committing current position. Under the new "cooperative" logic, we defer revocation until after the rebalance, which means we can continue fetching while the rebalance is in progress. However, when reviewing KAFKA-14196, we noticed that there is no similar logic to commit offsets prior to this deferred revocation. This means that cooperative consumption is more likely to lead to have duplicate consumption even when there is no failure involved. (was: With the old "eager" reassignment logic, we always revoked all partitions prior to each rebalance. When auto-commit is enabled, a part of this process is committing current position. Under the cooperative logic, we defer revocation until after the rebalance, which means we can continue fetching while the rebalance is in progress. However, when reviewing KAFKA-14196, we noticed that there is no similar logic to commit offsets prior to this deferred revocation. This means that cooperative consumption is more likely to lead to have duplicate consumption even when there is no failure involved.) > Consumer should auto-commit prior to cooperative partition revocation > - > > Key: KAFKA-14224 > URL: https://issues.apache.org/jira/browse/KAFKA-14224 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Major > > With the old "eager" reassignment logic, we always revoked all partitions > prior to each rebalance. When auto-commit is enabled, a part of this process > is committing current position. Under the new "cooperative" logic, we defer > revocation until after the rebalance, which means we can continue fetching > while the rebalance is in progress. However, when reviewing KAFKA-14196, we > noticed that there is no similar logic to commit offsets prior to this > deferred revocation. This means that cooperative consumption is more likely > to lead to have duplicate consumption even when there is no failure involved. -- This message was sent by Atlassian Jira (v8.20.10#820010)
