Hi Boynag, Yeah what you wrote is clear to me, that's why I asked this question.
What I don't understand if in the case of a producer we build a producer sourcing the data from whatever source (MySql for example) we need to take care explicitly of the mentioned idempotency per session problem. Do we need to take care of this, or is it already taken care of when using transactions ? On Fri, May 15, 2020 at 9:23 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > Hey Raffaele, > > the producer id is getting assigned upon receiving the > producer.initTransaction call at the broker side. It guarantees the > uniqueness of a producer for current lifecycle, which you don't have to > configure manually. > > Transactional API on the other hand, includes idempotent produce for sure. > You need to provide a unique transactional.id though. Feel free to learn > more high level stuff through this blog: > https://www.confluent.io/blog/transactions-apache-kafka/. > > Boyang > > Boyang > > On Fri, May 15, 2020 at 11:48 AM Raffaele Esposito <rafaelral...@gmail.com > > > wrote: > > > In relation to producer idempotency Kafka documentation says: > > > > > > ... Since each new instance of a producer is assigned a new, unique, PID, > > we can only guarantee idempotent production within a single producer > > session. > > > > > > Does it mean that when we build a producer sourcing the data from > whatever > > source (MySql for example) we need to take care explicitly of this > problem > > > > and put in place effective counter measures, like checking what was the > > last message actually produced successfully on Kafka ? > > > > > > What if we use transactions, even in the case we don't need really need > > transactional logic on writing messages to different partitions in an > all > > or nothing fashion ? > > > > Would using transactions in our producer get rid of the idempotency > issue ? > > > > > > > > Thanks! > > >
