Hello,
I try to understand a performance problem using Transactions:
Here my setup (for whatever reason, we need to create a transaction "per
message") (kafka v2.3/java):
-------------------
consumer.poll().forEach( record -> {
producer.beginTransaction();
producer.send(...);
...
producer.sendOffsetsToTransaction(...);
producer.commitTransaction();
}
-------------------
This setup is _really_ slow (difficult to do better than 20msg/s per
partitions). The brokers seem's not the bottlenek (cpu/io/network thread idle
... everything looks fine enough).
Could someone give me hints:
1) To some magic options to get better performances, client side or broker side
if needed ("batching" is not possible right now ... I am aware that it would be
though THE solution)
2) To try to understand where the performance is lost ? From my understanding,
it may be the lot of network round trip to achieve the Tx/send/commit/Tx ...
but I would like to really understand what's going on.
Thanks by advance for any answer !
Christophe
Post-scriptum La Poste
Ce message est confidentiel. Sous reserve de tout accord conclu par
ecrit entre vous et La Poste, son contenu ne represente en aucun cas un
engagement de la part de La Poste. Toute publication, utilisation ou diffusion,
meme partielle, doit etre autorisee prealablement. Si vous n'etes pas
destinataire de ce message, merci d'en avertir immediatement
l'expediteur.
