Hi Chris, Thanks for continuing to entertain some of these ideas.
On Fri, May 14, 2021 at 5:06 PM Chris Egerton <chr...@confluent.io.invalid> wrote: > [...] > That's true, but we do go from three static ACLs (write/describe on a fixed > transactional ID, and idempotent write on a fixed cluster) to a dynamic > collection of ACLs. > I'm not quite sure I follow, maybe I've lost track. To be clear, I was suggesting the use of a 'fencing producer' only in clusters with exactly.once.source.enabled=true where I imagined the key difference between the exactly once and fencing cases was how the producer was configured/used (transactional vs this new fencing semantic). I think the ACL requirements for connector producer principals would therefore be the same as currently described in the KIP. The same is true for the worker principals (which is the only breaking change you give in the KIP). So I don't think the fencing idea changes the backwards compatibility story that's already in the KIP, just allows a safe per-connector exactly.once=disabled option to be supported (with required as requested as we already discussed). But I'm wondering whether I've overlooked something. Ultimately I think it may behoove us to err on the side of reducing the > breaking changes here for now and saving them for 4.0 (or some later major > release), but would be interested in thoughts from you and others. > Difficult to answer (given I think I might be missing something). If there are breaking changes then I don't disagree. It's difficult to reason about big changes like this without some practical experience. If there are not, then I think we could also implement the whole exactly.once=disabled thing in a later KIP without additional breaking changes (i.e. some time in 3.x), right? > > Gouzhang also has a (possible) use case for a fencing-only producer ( > https://issues.apache.org/jira/browse/KAFKA-12693), and as he points out > there, you should be able to get these semantics today by calling > initTransactions() and then just using the producer as normal (no > beginTransaction()/abortTransaction()/endTransaction()). > > I tested this locally and was not met with success; transactional producers > do a check right now to ensure that any calls to "KafkaProducer::send" > occur within a transaction (see > > https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L957-L959 > and > > https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L450-L451 > ). > Not a blocker, just noting that we'd have to do some legwork to make this > workable with the producer API. > Ah, sorry, I should have actually tried it rather than just taking a quick look at the code. Rather than remove those safety checks I suppose we'd need a way of distinguishing, in the config, the difference in semantics. E.g. Something like a fencing.id config, which was mutually exclusive with transactional.id. Likewise perhaps initFencing() alongside initTransactions() in the API. But I think at this point it's something for another KIP. Kind regards, Tom
