When doing transaction, replication must finish broker side what implies increase latency. And the `commitTransaction` call is blocking (including a producer.flush()), the latency directly translates into decreased throughput.
-Matthias
On 3/12/20 2:40 AM, JALADY Christophe wrote:
> Hello,
>
> I try to understand a performance problem using Transactions:
>
> Here my setup (for whatever reason, we need to create a transaction "per
> message") (kafka v2.3/java):
>
> -------------------
> consumer.poll().forEach( record -> {
> producer.beginTransaction();
> producer.send(...);
> ...
> producer.sendOffsetsToTransaction(...);
> producer.commitTransaction();
> }
> -------------------
>
> This setup is _really_ slow (difficult to do better than 20msg/s per
> partitions). The brokers seem's not the bottlenek (cpu/io/network thread idle
> ... everything looks fine enough).
>
> Could someone give me hints:
>
> 1) To some magic options to get better performances, client side or broker
> side if needed ("batching" is not possible right now ... I am aware that it
> would be though THE solution)
>
> 2) To try to understand where the performance is lost ? From my
> understanding, it may be the lot of network round trip to achieve the
> Tx/send/commit/Tx ... but I would like to really understand what's going on.
>
>
> Thanks by advance for any answer !
>
>
> Christophe
>
>
> Post-scriptum La Poste
>
> Ce message est confidentiel. Sous reserve de tout accord conclu par
> ecrit entre vous et La Poste, son contenu ne represente en aucun cas un
> engagement de la part de La Poste. Toute publication, utilisation ou
> diffusion, meme partielle, doit etre autorisee prealablement. Si vous n'etes
> pas destinataire de ce message, merci d'en avertir immediatement
> l'expediteur.
>
signature.asc
Description: OpenPGP digital signature
