[ https://issues.apache.org/jira/browse/KAFKA-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liquan Pei reassigned KAFKA-3820: --------------------------------- Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Provide utilities for tracking source offsets > --------------------------------------------- > > Key: KAFKA-3820 > URL: https://issues.apache.org/jira/browse/KAFKA-3820 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect > Reporter: Ewen Cheslack-Postava > Assignee: Liquan Pei > > OffsetStorageReader does not (and is not expected to) be immediately updated > when a SourceRecord is returned from poll(). However, this can be a bit > confusing to connector developers as they may return that data, then expect a > subsequent read from OffsetStorageReader should match that. In other words, > rather than tracking which offset they are at themselves in variables > maintained by the task implementation, the connector developer expected > OffsetStorageReader to do this for them. > Part of the confusion comes from the fact that data is sent asynchronously > after returned from poll(), which explains the semantics we have. However, it > does also mean many connectors have similarly structured code where they keep > track of the current offset themselves. It might be nice to provide some > utilities, probably through the Context object, to get the last returned > offset for each source partition being processed by a task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)