yashmayya commented on PR #13465: URL: https://github.com/apache/kafka/pull/13465#issuecomment-1487350628
Thanks for the swift reply Chris and no rush on the review, I mainly wanted to get the clarifications regarding transactions on the offset topic(s). > Since we're introducing a new opt-in API, it's not a breaking change. That said, it's probably worth calling out in the KIP and on the discussion thread. And now that I think about it, maybe a transaction on the global offsets topic isn't really necessary if we've already done a round of zombie fencing. Thoughts? We should notify the discussion thread for the KIP no matter what, just wanted to bounce the idea off you first. Ah yeah that's a good point about the opt-in API, we wouldn't require the use of a transactional producer for regular connector task initiated offset writes to the worker's global offset backing store. However, I do agree that it doesn't seem necessary to use transactions on the global offsets topic if exactly-once source support is enabled and the connector is using a custom offsets topic (if it isn't, then we can write to the global offsets topic transactionally using a transactional producer corresponding to the connector). This will be in line with how regular offset writes are handled where not only do we not write offsets transactionally to the secondary store of a connector (the worker's global offset backing store assuming the connector has a custom offset topic configured as the primary store), but we also essentially ignore any errors arising from writes to the secondary store. > The use of transactions is only necessary if exactly-once source support is enabled for source connectors (both paragraphs that mention the use of transactions begin with "If exactly-once source support is enabled"). Ah okay, I misunderstood then. I thought the "If exactly-once source support is enabled" bit was only applicable to the zombie fencing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
