yashmayya commented on PR #13465: URL: https://github.com/apache/kafka/pull/13465#issuecomment-1486718538
Hi @C0urante, this will need to be rebased once https://github.com/apache/kafka/pull/13424 and https://github.com/apache/kafka/pull/13434 are merged but I raised this draft PR to solicit some early feedback if possible - particularly around this statement from the KIP: > Offsets will be reset transactionally for each topic that they exist in: a single transaction will be used to emit all tombstone records for the connector's dedicated offsets topic (if one is used) and another transaction will be used to emit all tombstone records for the worker's global offsets topic. We don't currently use a transactional producer for writing offsets to the worker's global offset backing store. Wouldn't doing so be a breaking change due to the requirement of additional ACLs for the worker's producer principal? For workers where exactly-once source support is enabled, the only way we could do so would be if the connector's configured offsets topic is the same as the worker's global offset topic (in which case, a connector specific offset store is used with a transactional producer). If the connector's configured offsets topic is a different one or if exactly-once source support is not enabled, I don't think we'll be able to write all the provided offsets in a single transaction for the worker's global offset backing store right? Also one minor comment on the KIP section - the "tombstone records" bit that was copied from the resetting offsets section needs to be updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
