Re:Re: open checkpoint, send message to kafka OutOfOrderSequenceException

[SmileSmile]

Hi,


 after failover still OutOfOrderSequenceException. when I close checkpoint, 
kafka broker still return OutOfOrderSequenceException to me .  







At 2021-06-04 17:52:22, "Yun Gao"  wrote:

Hi,



Have you checked if the error during normal execution, or right after failover?


Best,
Yun


--
From:SmileSmile 
Send Time:2021 Jun. 4 (Fri.) 11:07
To:user 
Subject:open checkpoint, send message to kafka OutOfOrderSequenceException


Dear all：
  flink version is 1.12.4，kafka version is 1.1.1。topology is very 
simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a few 
hours 。 the error message is 


Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: 
Failed to send data to Kafka: The broker received an out of order sequence 
number.
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392)
 
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850)
 
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99)
 


product's properties is simle ,  using defalut AT_LEAST_ONCE semantic
properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");

properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 
"1");
properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "2000");
properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, "524288");
properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "100");
properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 
"6");

My questions is how to deal with ？




 

Re: Re:Re: open checkpoint, send message to kafka OutOfOrderSequenceException

[Yun Gao]

Hi,

I tried google and found one possible related issue in [1]. Could you try to 
remove the config

properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5"); 

and try again ?


Best,
Yun

[1] 
https://stackoverflow.com/questions/55192852/transactional-producer-vs-just-idempotent-producer-java-exception-outoforderseq



 --Original Mail --
Sender:SmileSmile 
Send Date:Fri Jun 4 23:24:06 2021
Recipients:Yun Gao 
CC:user 
Subject:Re:Re: open checkpoint, send message to kafka 
OutOfOrderSequenceException



Hi,

 after failover still OutOfOrderSequenceException. when I close checkpoint, 
kafka broker still return OutOfOrderSequenceException to me .  


At 2021-06-04 17:52:22, "Yun Gao"  wrote:
Hi,

Have you checked if the error during normal execution, or right after failover?

Best,
Yun


--
From:SmileSmile 
Send Time:2021 Jun. 4 (Fri.) 11:07
To:user 
Subject:open checkpoint, send message to kafka OutOfOrderSequenceException

Dear all：  flink version is 1.12.4，kafka version is 1.1.1。topology is 
very simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a 
few hours 。 the error message is Caused by: 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send 
data to Kafka: The broker received an out of order sequence number.at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392)
 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850)
 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99)
 product's properties is simle ,  using defalut AT_LEAST_ONCE semantic
properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");  
  properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");
properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 
"1");properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 
"2000");properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, 
"524288");properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, 
"100");properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 
"6");My questions is how to deal with ？