It could work as you have described, but indeed only for one "local-only"
transaction. But that's not the case for me. It even wouldn't work with two
Kafka producers. If I could get away with just one single Kafka Producer I
wouldn't probably need distributed system to solve such use case.
Maybe
I'm not sure if I understood correctly, but if you want to integrate a
single kafka producer transaction (or any transaction manager that only
supports local transaction) into a distributed transaction, I think you
can do so as long as all other involved transaction managers support
2-phase
Sorry for responding to my own message, but when I sent an original
message/question I was not subscribed to this mailing list and now I can
not respond to Matthias answer directly.
I don't want to share a transaction between multiple Producers
threads/processes, I just would like to resume an
As Michał said. It's not designed for this use case.
Kafka's transaction, are not the same thing as DB transactions and if
you break it down, it allows for atomic (multi-partition) writes, but no
2-phase commit.
Also, a transaction is "owned" by a single thread (ie, producer) and
cannot be
But isn't it a low hanging fruit at this moment? Isn't that just an API
limitation and wouldn't the backend for transactions support it with only
minor changes to the API (do not fail automatically dangling transactions
on Producer restart)? Flushing is already there so that _should_ handle the
I don't think KIP-98 is as ambitious as to provide support for
distributed transactions (2 phase commit).
It would be great if I was wrong though :P
Cheers,
Michał
On 16/06/17 14:21, Piotr Nowojski wrote:
Hi,
I'm looking into Kafka's transactions API as proposed in KIP-98. I've read
both
Hi,
I'm looking into Kafka's transactions API as proposed in KIP-98. I've read
both this KIP-98 document and I looked into the code that is on the master
branch. I would like to use it to implement some two phase commit mechanism
on top of the Kafka's transactions, that would allow me to tie
