[
https://issues.apache.org/jira/browse/KAFKA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491626#comment-16491626
]
Manikumar commented on KAFKA-6947:
----------------------------------
By default mirror maker config property "abort.on.send.failure" is true. So
whenever producer gets error, mirror maker force close the producer instance.
"timeoutMillis = 0 ms" in logs indicates force closing of the producer.
We need to find out reason for producer send failure. you can try enabling
producer debug logs to for more information.
> Mirrormaker Closing producer due to send failure
> ------------------------------------------------
>
> Key: KAFKA-6947
> URL: https://issues.apache.org/jira/browse/KAFKA-6947
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.0
> Reporter: Andrew Holford
> Priority: Major
>
> Hi
> On occasion our mirrormakers fail with the below error....
> [2018-05-25 05:10:31,695] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 355 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 38 record(s) for
> com_snapshot--demo-5: 91886 ms has passed since last append
> [2018-05-25 05:10:31,710] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:31,710] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:31,710] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and just after this:
> [2018-05-25 05:10:32,041] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 323 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for
> com_snapshot--demo-5: 89429 ms has passed since batch creation plus linger
> time
> [2018-05-25 05:10:32,041] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,041] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,041] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and sometimes we get:
> [2018-05-25 08:24:29,249] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 354 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:183)
> at java.lang.Thread.run(Thread.java:745)
> [2018-05-25 08:24:29,249] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 08:24:29,249] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 08:24:29,249] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> then it completely shuts down:
> [2018-05-25 05:10:32,060] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,083] INFO [mirrormaker-thread-0] Flushing producer.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO [mirrormaker-thread-0] Committing consumer
> offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,105] INFO [mirrormaker-thread-0] Shutting down consumer
> connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,138] INFO Publish thread interrupted!
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,144] INFO Publishing Monitoring Metrics stopped for
> clientID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,148] INFO Closing the Kafka producer with timeoutMillis
> = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,183] INFO Closed monitoring interceptor for client
> ID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,197] INFO [mirrormaker-thread-0] Mirror maker thread
> stopped (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,202] FATAL [mirrormaker-thread-0] Mirror maker thread
> exited abnormally, stopping the whole mirror maker.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,203] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,245] INFO Start clean shutdown.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,245] INFO Shutting down consumer threads.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,261] INFO [mirrormaker-thread-0] mirrormaker-thread-0
> shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,265] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> On the broker side we see:
> WARN Attempting to send response via channel for which there is no open
> connection, connection id 10.82.6.105:9093-172.27.205.216:32796
> (kafka.network.Processor)
> Does anyone know what could cause this and what a possible solution could be?
> Im a little confused by the "timeoutMillis = 0 ms" as well mentioned, is this
> some setting which needs adjusting somewhere? We have
> request.timeout.ms=60000 on the producer config with most other settings left
> as the defaults.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
