[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225215#comment-14225215 ] Patrick Lucas commented on KAFKA-1273: -- I ran into this on 0.8.1.1. Perhaps if you override message.max.bytes in the config file it complains on startup, but Kafka allows you to modify this value per-topic. I have a topic with a higher value of message.max.bytes, and it was very difficult to diagnose why its replicas could not stay in sync. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225322#comment-14225322 ] Sriharsha Chintalapani commented on KAFKA-1273: --- [~nehanarkhede] Yes. working on reproducing it. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226570#comment-14226570 ] Sriharsha Chintalapani commented on KAFKA-1273: --- Thanks @Patrick Lucas. I am able to reproduce this . [~nehanarkhede] I don't think I've permissions to re-open the JIRA. Could you please re-open , I'll send a patch. Thanks. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226851#comment-14226851 ] Sriharsha Chintalapani commented on KAFKA-1273: --- [~nehanarkhede] this doesn't look straight forward as TopicCommand doesn't know what values are in server.properties. It calls LogConfig to validate which doesn't have access to this info. One way of doing this is to pass server.properties to the TopicCommand which doesn't look intuitive. Another option is to add an api that return a configResponse which contains all the server.properties that user defined along with defaults. Please let me know what you think about these approaches. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226929#comment-14226929 ] Neha Narkhede commented on KAFKA-1273: -- I'm not able to reopen the issue as well. [~junrao], are you able to? > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232393#comment-14232393 ] Jun Rao commented on KAFKA-1273: Kafka jira is configured with no reopen after close. We can probably just create a new jira. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232479#comment-14232479 ] Sriharsha Chintalapani commented on KAFKA-1273: --- [~nehanarkhede] [~junrao] Any thoughts on proposed approach adding an api to return broker config > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244556#comment-14244556 ] Jun Rao commented on KAFKA-1273: I was thinking that one way is to just disallow customizing max.message.size per topic. Such customization may break downstream consumers like MirrorMaker. I am not sure if there is a strong use case for the per topic customization. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Sriharsha Chintalapani > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1273) Brokers should make sure replica.fetch.max.bytes >= message.max.bytes
[ https://issues.apache.org/jira/browse/KAFKA-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990162#comment-13990162 ] sriharsha chintalapani commented on KAFKA-1273: --- I am not able to reproduce this in the trunk as kafka-server-start.sh fails to start the broker throwing java.lang.IllegalArgumentException: requirement failed: replica.fetch.max.bytes has value 1024 which is not in the range (2048,2147483647). its checking if the replica.fetch.max.bytes >= message.max.bytes. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes > - > > Key: KAFKA-1273 > URL: https://issues.apache.org/jira/browse/KAFKA-1273 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Dong Zhong >Assignee: Neha Narkhede > Labels: newbie > > If message.max.bytes is larger than replica.fetch.max.bytes，followers can't > fetch data from the leader and will incur endless retry. And this may cause > high network traffic between followers and leaders. > Brokers should make sure replica.fetch.max.bytes >= message.max.bytes by > adding a sanity check, or throw an exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
