I've been pondering this question of coordinating other resource managers with Kafka transactions for a while and I'm not convinced it's a good idea. My reservations come down to the guarantees that it would provide in failure scenarios.
I don't think KIP-98 gives proper ACID semantics in the presence of all failures. For a transaction which contains a mixture of publishes and offset updates, a bunch of topics are involved and it appears to me that an uncontrolled shutdown could result in some but not all of the lazy writes making it to disk. Here are some of the failures that I'm worried about: * A message is published to a topic which crashes the leader Kafka node, as it's replicated across the cluster, it crashes all of the other Kafka nodes (we've really had this - SEGV, our fault and we've fixed it, but it happened) so this is a kind of rolling node crash in a cluster * Out of memory error in one or more Kafka nodes * Disk fills in one or more Kafka nodes * Uncontrolled power-off to all nodes in the cluster Does KIP-98 guarantee atomicity for transactions in all of these cases? Unless all of the topics involved in a transaction are recovered to the same point in time, you can't consider a transaction to be properly atomic. If application code is designed expecting atomicity, there are going to be tears. Perhaps only when disaster strikes, but the risk is there. I think KIP-98 is interesting, but I wouldn't equate what it provides with the transactions provided by resource managers with traditional transaction logging. It's not better or worse, just different. If you tried to migrate from a previous transactional system to Kafka transactions, I think you'd better have procedures for reconciliation with the other resource managers. Better still, don't build applications that are so fragile. The principle of dumb pipes and smart endpoints is good in my view. If you're creating a global transaction containing two or more resource managers and using two-phase commit, it's very important that all of the resource managers maintain a coherent view of the sequence of events. If any part fails due to a software or hardware failure, once the system is recovered, nothing must be forgotten. If you look at how presume-abort works, you'll see how important this is. Kafka doesn't really fit very nicely in this kind of environment because of the way that it writes lazily to disk. The theory is that you must avoid at all costs having an uncontrolled shutdown of an entire cluster because you'll lose a little data at the end of the logs. So, if you are coordinating Kafka and a relational database in a global transaction, it's theoretically possible that a crashed Kafka would be a little forgetful while a crashed database would not. The database would be an order of magnitude or more slower because of the way its recovery logs are handled, but it would not be forgetful in the same way. You get exactly the same kind of worries when you implement some kind of asynchronous replication for disaster recovery, even if all of the resource managers force all of their log writes to disk eagerly. The replica at the DR site is slightly behind the primary site, so if you have to recover from an outage and switch to the DR site, it can be considered to be slightly forgetful about the last few moments before the outage. This is why a DR plan usually has some concept of compensation or reconciliation to make good any forgotten work. In summary, I think Kafka would have to change in ways which would negate many of its good points in order to support XA transactions. It would be better to design applications to be resilient to message duplication and loss, rather than tightly coupling resource managers and ending up with something fragile. Don't get me wrong. This is not an anti-Kafka rant. I just work with people used to traditional transactional systems, making use of Kafka for business applications, and it's important to understand the concepts on both sides and make sure your assumptions are valid. Andrew Schofield IBM Watson and Cloud Platform > From: Michael Pearce <michael.pea...@ig.com> > Sent: 09 December 2016 06:19 > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional > Messaging > > Usecase in IG: > > Fund transfer between accounts. When we debit one account and fund another we > must ensure the records to both occur > as an acid action, and as a single > transaction. > > Today we achieve this because we have jms, as such we can do the actions > needed in an xa transaction across both the > accounts. To move this flow to > Kafka we would need support of XA transaction. > > > > Sent using OWA for iPhone > ________________________________________ > From: Michael Pearce <michael.pea...@ig.com> > Sent: Friday, December 9, 2016 6:09:06 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional > Messaging > > Hi Jay, > > For me having an XA transaction allows for ensuring ACID across my > application. > > I believe it is part of the JMS api, and obviously JMS still is in enterprise > very widely adopted for Messaging transport , so > obviously to say it isn't > widely used i think is ignoring a whole range of users. Like wise I believe > frameworks like spring etc fully support it more evidence of its wide > adoption. > > On this note personally we try to avoid transactions entirely in our flows > for performance and simplicity. but we do alas unfortunately have one or two > places we cannot ignore it. > > Cheers > Mike > > Sent using OWA for iPhone > ________________________________________ > From: Jay Kreps <j...@confluent.io> > Sent: Thursday, December 8, 2016 11:25:53 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional > Messaging > > Hey Edoardo, > > For (3) can you outline what you think the benefit and use cases for a more > general cross-system XA feature would be an what changes to the proposal > would be required to enable it? When I have asked people who wanted > cross-system XA in the past what they wanted it for, I haven't really > gotten any answers that made sense. Every person really wanted something > that would be better solved by a transactional (or idempotent) write to > Kafka followed by an independent transactional (or idempotent) consumption > (which this proposal enables). For the use cases they described tying these > two things together had no advantage and many disadvantages. > > I have one use case which would be accomplished by cross-system XA which is > allowing the producer to block on the synchronous processing of the message > by (all? some?) consumers. However I'm not convinced that cross-system XA > is the best solution to this problem, and I'm also not convinced this is an > important problem to solve. But maybe you have something in mind here. > > -Jay > > > > On Thu, Dec 8, 2016 at 1:15 PM, Edoardo Comar <eco...@uk.ibm.com> wrote: > > > Hi, > > thanks, very interesting KIP ... I haven't fully digested it yet. > > > > We have many users who choose not to use the Java client, so I have > > concerns about the added complexity in developing the clients. > > A few questions. > > > > 1 - is mixing transactional and non transactional messages on the *same > > topic-partition* really a requirement ? > > What use case does it satisfy? > > > > 2 - I guess some clients may only be interested to implement the producer > > idempotency. > > It's not clear how they could be implemented without having to add the > > transaction capabilities. > > As others on this list have said, I too would like to see idempotency as a > > more basic feature, on top which txns can be built. > > > > 3 - The KIP seems focused on a use case where consumption from a topic and > > subsequent production are part of the producer transaction. > > > > It'd be great to see a way to extend the producer transaction to include > > additional transactional resources, > > so that the consumption from another topic just becomes a special case of > > a more general "distributed" txn. > > > > Edo > > -------------------------------------------------- > > Edoardo Comar > > IBM MessageHub > > eco...@uk.ibm.com > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > IBM United Kingdom Limited Registered in England and Wales with number > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 > 3AU > > > > From: Guozhang Wang <wangg...@gmail.com> > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > Date: 30/11/2016 22:20 > Subject: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional > Messaging > > > > Hi all, > > I have just created KIP-98 to enhance Kafka with exactly once delivery > semantics: > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > >* > > This KIP adds a transactional messaging mechanism along with an idempotent > producer implementation to make sure that 1) duplicated messages sent from > the same identified producer can be detected on the broker side, and 2) a > group of messages sent within a transaction will atomically be either > reflected and fetchable to consumers or not as a whole. > > The above wiki page provides a high-level view of the proposed changes as > well as summarized guarantees. Initial draft of the detailed > implementation > design is described in this Google doc: > > https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF > 0wSw9ra8 > > > We would love to hear your comments and suggestions. > > Thanks, > > -- Guozhang > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
