Hi there,
Finally we enabled idempotence, and setting as per official documentation
max.in.flight.requests.per.connection <= 5 and acks = all.
'enable.idempotence' => 'true',
'max.in.flight.requests.per.connection' => '4',
'acks' => 'all',
" When set to 'true', the producer will ensure that exactly one copy of each
message is written in the stream. If 'false', producer retries due to broker
failures, etc., may write duplicates of the retried message in the stream. Note
that enabling idempotence requires max.in.flight.requests.per.connection to be
less than or equal to 5, retries to be greater than 0 and acks must be 'all'.
If these values are not explicitly set by the user, suitable values will be
chosen. If incompatible values are set, a ConfigException will be thrown."
And now facing problems to create producer, forcing
max.in.flight.requests.per.connection = 1:
Caused by: org.apache.kafka.common.config.ConfigException: Must set
max.in.flight.requests.per.connection to 1 in orderto use the idempotent
producer. Otherwise we cannot guarantee idempotence.
at
org.apache.kafka.clients.producer.KafkaProducer.configureInflightRequests(KafkaProducer.java:482)
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:360)
... 17 common frames omitted
Any idea about?
Our Kafka broker version is 2.2.1.
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: [email protected]
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 29/08/2019, 17:47, "Anatoliy Soldatov" <[email protected]> wrote:
What exactly is the problem?
Kafka states it in documentation and it is not new.
Quote from docs: max.in.flight.requests.per.connection – The maximum number
of unacknowledged requests the client will send on a single connection before
blocking. Note that if this setting is set to be greater than 1 and there are
failed sends, there is a risk of message re-ordering due to retries (i.e., if
retries are enabled).
The possible problem is only with some people not able to read
documentation.
29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy
<[email protected]<mailto:[email protected]>> написал(а):
Hi there,
Recently we found this blog entry about a possible problem with messages
order:
https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.softwaremill.com_does-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=eoZO2gqSx3WfGCIIcane_DH28EY2j1xHy5pwK9o70L4&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fblog.softwaremill.com-252Fdoes-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034629468-26sdata-3D-252BulzkEeXQ6xhczBYfYQ8wnvWrpW-252Fs3wVtQaGGwIgPbM-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=dFwxCXAt09CuEx1CL20C8BR4SKDsWzZgfdIQ3U6ZAi8&e=
>
Please could you confirm about? And if so, how to fix? Blog entry
suggesting how to.
Thanks
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ph2WXQKWT4O1uhip6cQzhhNHSKnxw6S5xChr13VQmMk&e=
]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.williamhill.com-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034639463-26sdata-3DKowmt6DYcYKQgMC-252FI5Yv-252BeZNsaPhh8oK8-252F9jpvfyuSk-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=T7tzoiDU2XkOMiQ_Hq0KEBKH3DoW7lM2Hv2jGqwutjc&e=
>
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_senet.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=_o5y3RjRJ1q8K5sKGbyP0-Ryn19BzNYT9gG16G_BDO8&e=
]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.whenthefunstops.co.uk-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034649456-26sdata-3D-252FtXL-252FsYYwa2yGdPRNQ7-252FpYAos0CkWYxaoVqwLJP0Xfs-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ezMzWlYi-DwoK2lzhJC3AQltVduPff9GdL51lWI5wSk&e=
>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: [email protected]<mailto:[email protected]>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
________________________________
"This message contains confidential information/commercial secret. If you
are not the intended addressee of this message you may not copy, save, print or
forward it to any third party and you are kindly requested to destroy this
message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию,
являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом
данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать
его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об
этом отправителя электронным письмом.”
