[ https://issues.apache.org/jira/browse/KAFKA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491626#comment-16491626 ]
Manikumar commented on KAFKA-6947: ---------------------------------- By default mirror maker config property "abort.on.send.failure" is true. So whenever producer gets error, mirror maker force close the producer instance. "timeoutMillis = 0 ms" in logs indicates force closing of the producer. We need to find out reason for producer send failure. you can try enabling producer debug logs to for more information. > Mirrormaker Closing producer due to send failure > ------------------------------------------------ > > Key: KAFKA-6947 > URL: https://issues.apache.org/jira/browse/KAFKA-6947 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.11.0.0 > Reporter: Andrew Holford > Priority: Major > > Hi > On occasion our mirrormakers fail with the below error.... > [2018-05-25 05:10:31,695] ERROR Error when sending message to topic > com_snapshot--demo with key: 13 bytes, value: 355 bytes with error: > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > org.apache.kafka.common.errors.TimeoutException: Expiring 38 record(s) for > com_snapshot--demo-5: 91886 ms has passed since last append > [2018-05-25 05:10:31,710] INFO Closing producer due to send failure. > (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:31,710] INFO Closing the Kafka producer with timeoutMillis > = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer) > [2018-05-25 05:10:31,710] INFO Proceeding to force close the producer since > pending requests could not be completed within timeout 0 ms. > (org.apache.kafka.clients.producer.KafkaProducer) > and just after this: > [2018-05-25 05:10:32,041] ERROR Error when sending message to topic > com_snapshot--demo with key: 13 bytes, value: 323 bytes with error: > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for > com_snapshot--demo-5: 89429 ms has passed since batch creation plus linger > time > [2018-05-25 05:10:32,041] INFO Closing producer due to send failure. > (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:32,041] INFO Closing the Kafka producer with timeoutMillis > = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer) > [2018-05-25 05:10:32,041] INFO Proceeding to force close the producer since > pending requests could not be completed within timeout 0 ms. > (org.apache.kafka.clients.producer.KafkaProducer) > and sometimes we get: > [2018-05-25 08:24:29,249] ERROR Error when sending message to topic > com_snapshot--demo with key: 13 bytes, value: 354 bytes with error: > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > java.lang.IllegalStateException: Producer is closed forcefully. > at > org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609) > at > org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596) > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:183) > at java.lang.Thread.run(Thread.java:745) > [2018-05-25 08:24:29,249] INFO Closing producer due to send failure. > (kafka.tools.MirrorMaker$) > [2018-05-25 08:24:29,249] INFO Closing the Kafka producer with timeoutMillis > = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer) > [2018-05-25 08:24:29,249] INFO Proceeding to force close the producer since > pending requests could not be completed within timeout 0 ms. > (org.apache.kafka.clients.producer.KafkaProducer) > then it completely shuts down: > [2018-05-25 05:10:32,060] INFO Exiting on send failure, skip committing > offsets. (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:32,083] INFO [mirrormaker-thread-0] Flushing producer. > (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,094] INFO [mirrormaker-thread-0] Committing consumer > offsets. (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,094] INFO Exiting on send failure, skip committing > offsets. (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:32,105] INFO [mirrormaker-thread-0] Shutting down consumer > connectors. (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,138] INFO Publish thread interrupted! > (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor) > [2018-05-25 05:10:32,144] INFO Publishing Monitoring Metrics stopped for > clientID=peach-prod2prod3-mirrormaker-0 > (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor) > [2018-05-25 05:10:32,148] INFO Closing the Kafka producer with timeoutMillis > = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer) > [2018-05-25 05:10:32,183] INFO Closed monitoring interceptor for client > ID=peach-prod2prod3-mirrormaker-0 > (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor) > [2018-05-25 05:10:32,197] INFO [mirrormaker-thread-0] Mirror maker thread > stopped (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,202] FATAL [mirrormaker-thread-0] Mirror maker thread > exited abnormally, stopping the whole mirror maker. > (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,203] INFO [mirrormaker-thread-0] Mirror maker thread > shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,245] INFO Start clean shutdown. > (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:32,245] INFO Shutting down consumer threads. > (kafka.tools.MirrorMaker$) > [2018-05-25 05:10:32,261] INFO [mirrormaker-thread-0] mirrormaker-thread-0 > shutting down (kafka.tools.MirrorMaker$MirrorMakerThread) > [2018-05-25 05:10:32,265] INFO [mirrormaker-thread-0] Mirror maker thread > shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread) > On the broker side we see: > WARN Attempting to send response via channel for which there is no open > connection, connection id 10.82.6.105:9093-172.27.205.216:32796 > (kafka.network.Processor) > Does anyone know what could cause this and what a possible solution could be? > Im a little confused by the "timeoutMillis = 0 ms" as well mentioned, is this > some setting which needs adjusting somewhere? We have > request.timeout.ms=60000 on the producer config with most other settings left > as the defaults. -- This message was sent by Atlassian JIRA (v7.6.3#76005)