0 <https://stackoverflow.com/posts/72491139/timeline>
I have an issue with kafka producer in production where I see below error. *"Publish failed, Expiring 2 record(s) for myTopic-1:120004 ms has passed since batch creation[[ org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for myTopic-1:120004 ms has passed since batch creation"* Kafka brokers are of confluent 5.3.2 version and the kafka-client is apache 2.3.1. Producer config which are explicitly specified in my code are below and remaining are defaults. batch.size = 102400 linger.ms = 100 compression.type = lz4 ack = all *Sample Java Code* ProducerRecord<String, String> rec = new ProducerRecord<String, String>("myTopic,1,"myKey","json-payload-here"); producer.send(rec, new ProducerCallback(jsonPayload)); private class ProducerCallback implements Callback { private String _ME ="onCompletion"; private String jsonPayload; public ProducerCallback(String jsonPayload) { this.jsonPayload = jsonPayload; } @Override public void onCompletion(RecordMetadata recordMetadata, Exception e) { if (e == null) { LOG.logp(Level.FINEST, _CL, _ME, "Published kafka event "+jsonPayload); } else { //Note: Exception is logged here. LOG.log(Level.SEVERE, "Publish failed, "+e.getMessage(), e); } } } *Couple of questions* 1. Load is not heavy in production and its moderate as of now and might be heavy in later stages. Am I missing some producer config to rectify above issue? 2. Assuming 2 records has been expired in the batch, Is there a way I can get those expired records in java so that I can get payload and key to republish them? Thanks, appreciate your help in advance.