[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738593#comment-16738593
]
Joseph Niemiec commented on KAFKA-6706:
---
Wanted to follow up here that we upgraded one of our production clusters from
1.0.0 to 1.1.1 and like David above this appears to have resolved the issue.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-18, retrying (2147483646 attempts
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718052#comment-16718052
]
David van Geest commented on KAFKA-6706:
FWIW upgrading to Kafka 1.1.1 solved these problems for us.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.781 [kafka-producer-network-thread |
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718049#comment-16718049
]
Joseph Niemiec commented on KAFKA-6706:
---
We have all the same error messages occurring in our production clusters. It
started after we upgraded from 0.8 to 1.0.0.
{code:java}
// 2018-12-11 16:40:43 DEBUG NetworkClient:183 - [Producer
clientId=KafkaExampleProducer] Disconnecting from node 1 due to request timeout.
2018-12-11 16:40:43 WARN Sender:251 - [Producer clientId=KafkaExampleProducer]
Got error produce response with correlation id 21193 on topic-partition
debug_dev_r2k-3, retrying (4 attempts left). Error: REQUEST_TIMED_OUT
2018-12-11 16:40:43 WARN Sender:251 - [Producer clientId=KafkaExampleProducer]
Got error produce response with correlation id 21203 on topic-partition
debug_dev_r2k-0, retrying (4 attempts left). Error: NETWORK_EXCEPTION
2018-12-11 16:40:43 WARN Sender:251 - [Producer clientId=KafkaExampleProducer]
Got error produce response with correlation id 21203 on topic-partition
debug_dev_r2k-3, retrying (4 attempts left). Error: NETWORK_EXCEPTION
2018-12-11 16:40:43 WARN Sender:251 - [Producer clientId=KafkaExampleProducer]
Got error produce response with correlation id 21203 on topic-partition
debug_dev_r2k-6, retrying (4 attempts left). Error: NETWORK_EXCEPTION
{code}
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586534#comment-16586534
]
David van Geest commented on KAFKA-6706:
We also noticed this regression once we updated to 1.0.1 from 0.11.0.1. Not
MirrorMaker in our case, but a number of different apps using a variety of
clients (definitely the official Java client, but others as well).
We are currently testing 1.1.1, and it does not seem to have the same problems.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427950#comment-16427950
]
Di Shang commented on KAFKA-6706:
-
Upgrading broker to Kafka 1.1.0 seems to resolve the issue (haven't seen those
exceptions so far), I looked through the release notes but can't seem to find
anything immediately obvious that might be related.
I will monitor our clusters for a little while to make sure it is truely fixed.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
>
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421904#comment-16421904
]
Paul Lin commented on KAFKA-6706:
-
I'm hitting the same issue in a performance test against Kafka 1.0.0. With a
topic of 6+ partitions and 3 replicas, the producer would get lots of network
exceptions when setting 'acks=all'. I assumed that there're some performance
problems with the server, because the server side log shows that when the
server tries to response to the client, the connection is already closed by the
other side.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Blocker
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread |
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413355#comment-16413355
]
Di Shang commented on KAFKA-6706:
-
@[~sunzhenya] I have read through KIP-91 before but it's not even released yet.
Also as I described, this is more likely a server side issue caused by the
broker upgrade, downgrading server to 0.10.2 resolves it while using the same
client and config.
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Major
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with
[jira] [Commented] (KAFKA-6706) NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0 broker upgrade
[
https://issues.apache.org/jira/browse/KAFKA-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410938#comment-16410938
]
abel-sun commented on KAFKA-6706:
-
If you haven't please check KIP-91,The hope can help you!
https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer
> NETWORK_EXCEPTION and REQUEST_TIMED_OUT in mirrormaker producer after 1.0
> broker upgrade
>
>
> Key: KAFKA-6706
> URL: https://issues.apache.org/jira/browse/KAFKA-6706
> Project: Kafka
> Issue Type: Bug
> Components: core, network
>Affects Versions: 1.0.0
>Reporter: Di Shang
>Priority: Major
> Labels: mirror-maker
>
> We have 2 clusters A and B with 4 brokers each, we use mirrormaker to
> replicate topics from A to B.
> We recently upgraded our brokers from 0.10.2.0 to 1.0.0, after the upgrade
> we started seeing the mirrormaker task showing producer errors and
> intermittently dying.
> We tried using 1.0.0 and 0.10.2.0 mirrormaker, both have the same problem.
> Downgrading cluster B brokers back to 0.10.2.0 and the problem went away, so
> we think it's a server side problem.
> There are 2 types of errors: REQUEST_TIMED_OUT and NETWORK_EXCEPTION. For
> testing, I used a topic *logging* with 20 partitions and 3 replicas (same on
> cluster A and B), the source topic has 50+ million msg.
> (this is from mirrormaker 1.0 at info level, the 0.10.2.0 log is very similar)
> {noformat}
> 22 Mar 2018 02:16:07.407 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 35122 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:17:49.731 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 51572 on
> topic-partition logging-7, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:18:33.903 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 57785 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:21:21.399 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 85406 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:25:22.278 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 128047 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:26:17.154 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 137049 on
> topic-partition logging-18, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.358 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 153976 on
> topic-partition logging-5, retrying (2147483646 attempts left). Error:
> REQUEST_TIMED_OUT
> 22 Mar 2018 02:27:57.779 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-2, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-10, retrying (2147483646 attempts left). Error:
> NETWORK_EXCEPTION
> 22 Mar 2018 02:27:57.780 [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.producer.internals.Sender warn(line:251) [Producer
> clientId=producer-1] Got error produce response with correlation id 154077 on
> topic-partition logging-18, retrying (2147483646 attempts left).
