[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Resolution: Pending Closed Status: Resolved (was: Patch Available) Committed to trunk. > Offsets manager cache should prevent stale-offset-cleanup while an offset > load is in progress; otherwise we can lose consumer offsets > - > > Key: KAFKA-2163 > URL: https://issues.apache.org/jira/browse/KAFKA-2163 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Fix For: 0.8.3 > > Attachments: KAFKA-2163.patch > > > When leadership of an offsets partition moves, the new leader loads offsets > from that partition into the offset manager cache. > Independently, the offset manager has a periodic cleanup task for stale > offsets that removes old offsets from the cache and appends tombstones for > those. If the partition happens to contain much older offsets (earlier in the > log) and inserts those into the cache; the cleanup task may run and see those > offsets (which it deems to be stale) and proceeds to remove from the cache > and append a tombstone to the end of the log. The tombstone will override the > true latest offset and a subsequent offset fetch request will return no > offset. > We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2163: --- Reviewer: Jun Rao > Offsets manager cache should prevent stale-offset-cleanup while an offset > load is in progress; otherwise we can lose consumer offsets > - > > Key: KAFKA-2163 > URL: https://issues.apache.org/jira/browse/KAFKA-2163 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Fix For: 0.8.3 > > Attachments: KAFKA-2163.patch > > > When leadership of an offsets partition moves, the new leader loads offsets > from that partition into the offset manager cache. > Independently, the offset manager has a periodic cleanup task for stale > offsets that removes old offsets from the cache and appends tombstones for > those. If the partition happens to contain much older offsets (earlier in the > log) and inserts those into the cache; the cleanup task may run and see those > offsets (which it deems to be stale) and proceeds to remove from the cache > and append a tombstone to the end of the log. The tombstone will override the > true latest offset and a subsequent offset fetch request will return no > offset. > We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Attachment: KAFKA-2163.patch > Offsets manager cache should prevent stale-offset-cleanup while an offset > load is in progress; otherwise we can lose consumer offsets > - > > Key: KAFKA-2163 > URL: https://issues.apache.org/jira/browse/KAFKA-2163 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy > Fix For: 0.8.3 > > Attachments: KAFKA-2163.patch > > > When leadership of an offsets partition moves, the new leader loads offsets > from that partition into the offset manager cache. > Independently, the offset manager has a periodic cleanup task for stale > offsets that removes old offsets from the cache and appends tombstones for > those. If the partition happens to contain much older offsets (earlier in the > log) and inserts those into the cache; the cleanup task may run and see those > offsets (which it deems to be stale) and proceeds to remove from the cache > and append a tombstone to the end of the log. The tombstone will override the > true latest offset and a subsequent offset fetch request will return no > offset. > We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Assignee: Joel Koshy Status: Patch Available (was: Open) > Offsets manager cache should prevent stale-offset-cleanup while an offset > load is in progress; otherwise we can lose consumer offsets > - > > Key: KAFKA-2163 > URL: https://issues.apache.org/jira/browse/KAFKA-2163 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Fix For: 0.8.3 > > Attachments: KAFKA-2163.patch > > > When leadership of an offsets partition moves, the new leader loads offsets > from that partition into the offset manager cache. > Independently, the offset manager has a periodic cleanup task for stale > offsets that removes old offsets from the cache and appends tombstones for > those. If the partition happens to contain much older offsets (earlier in the > log) and inserts those into the cache; the cleanup task may run and see those > offsets (which it deems to be stale) and proceeds to remove from the cache > and append a tombstone to the end of the log. The tombstone will override the > true latest offset and a subsequent offset fetch request will return no > offset. > We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
