[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Resolution: Pending Closed Status: Resolved (was: Patch Available) Committed to trunk. Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets - Key: KAFKA-2163 URL: https://issues.apache.org/jira/browse/KAFKA-2163 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.3 Attachments: KAFKA-2163.patch When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache. Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset. We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2163: --- Reviewer: Jun Rao Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets - Key: KAFKA-2163 URL: https://issues.apache.org/jira/browse/KAFKA-2163 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.3 Attachments: KAFKA-2163.patch When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache. Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset. We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Attachment: KAFKA-2163.patch Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets - Key: KAFKA-2163 URL: https://issues.apache.org/jira/browse/KAFKA-2163 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Fix For: 0.8.3 Attachments: KAFKA-2163.patch When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache. Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset. We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Assignee: Joel Koshy Status: Patch Available (was: Open) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets - Key: KAFKA-2163 URL: https://issues.apache.org/jira/browse/KAFKA-2163 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.3 Attachments: KAFKA-2163.patch When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache. Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset. We just need to prevent the cleanup task from running during an offset load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
