[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964407#comment-14964407 ] ASF GitHub Bot commented on KAFKA-2515: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/332 > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Guozhang Wang >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964358#comment-14964358 ] ASF GitHub Bot commented on KAFKA-2515: --- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/332 HOTFIX: check logic of KAFKA-2515 should be on buffer.limit() You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K2515-hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/332.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #332 commit 76c4bde6e3e7c0d8e816f5d4393d231a91f34185 Author: Guozhang Wang Date: 2015-10-20T01:27:38Z KAFKA-2515.hf > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Guozhang Wang >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961587#comment-14961587 ] ASF GitHub Bot commented on KAFKA-2515: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/318 > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Guozhang Wang >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961064#comment-14961064 ] Guozhang Wang commented on KAFKA-2515: -- Yup, in the latest patch we will throw RecordTooLarge in poll() calls. > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Guozhang Wang >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960997#comment-14960997 ] Onur Karaman commented on KAFKA-2515: - +1 on Jason's comment. With just the log message, it seems like it would be harder to react to the large message programmatically. What does ZookeeperConsumerConnector do? > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Guozhang Wang >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959414#comment-14959414 ] Jason Gustafson commented on KAFKA-2515: I'm a little unclear why we'd only want to log this error. As a user, I would probably rather receive an exception notifying me when this happens. Then I can choose to seek past the message or propagate the exception. > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959387#comment-14959387 ] Guozhang Wang commented on KAFKA-2515: -- I have verified following the above steps that with this patch oversided messages will cause an error message; and since we keep retrying it will iteratively log error messages until users shut it down. My question is do we want to close the consumer actively upon encountering this issue moving forward? > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959379#comment-14959379 ] ASF GitHub Bot commented on KAFKA-2515: --- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/318 KAFKA-2515: Handle oversized messages properly in new consumer You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K2515 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/318.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #318 commit 8d5cafabc35f69b53c204101aad6014688ac38dd Author: Guozhang Wang Date: 2015-10-15T01:16:27Z KAFKA-2515.v1 commit 4f991a1039789a192e2cc0ed1368344a37be4fb1 Author: Guozhang Wang Date: 2015-10-15T18:38:09Z remove comments > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959035#comment-14959035 ] Jun Rao commented on KAFKA-2515: [~guozhang], right. > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958099#comment-14958099 ] Guozhang Wang commented on KAFKA-2515: -- [~junrao] just to clarify, for step 3) above you have --new-consumer enabled in console consumer right? > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman >Priority: Blocker > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952094#comment-14952094 ] Jun Rao commented on KAFKA-2515: The following is how you can reproduce this. 1. Create a new topic test3, allowing a larger message size. kafka-topics --zookeeper localhost:2181 --create --topic test3 --partition 1 --replication-factor 1 --config max.message.bytes=130 2. Add one oversize message. kafka-run-class org.apache.kafka.clients.tools.ProducerPerformance test3 1 120 1000 bootstrap.servers=localhost:9090,localhost:9091,localhost:9092 max.request.size=200 3. Consume messages on topic test3. kafka-console-consumer --from-beginning --topic test3 --zookeeper localhost:2181 > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744252#comment-14744252 ] Edward Ribeiro commented on KAFKA-2515: --- Ewen was able to simulate the bug with a simple config with two brokers, adjusting message.max.bytes and replica.fetch.max.bytes. Then he created a topic with replication factor 2 and used console producer to send data of different sizes to test the output. Unfortunately KAFKA-2338 has been on hold for the past two months so I am back to square 1. :( Gonna try to repro here following Ewen's strategy, but let's keep in touch. Regards, > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744151#comment-14744151 ] Onur Karaman commented on KAFKA-2515: - Hi [~eribeiro]. To be honest, I just started looking at this ticket. The two might be related. Just curious, do you already know how to reproduce the issue? If not, it's no big deal and I can look into it. > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744102#comment-14744102 ] Edward Ribeiro commented on KAFKA-2515: --- [~onurkaraman], isn't this issue related to KAFKA-2338? I mean, I would have to modify the consumer for address it, but I can limit myself to logging on the server side. But it would be cool to exchange ideas about this oversized message problem, so I would suggest to link those two issues. Cheers! > handle oversized messages properly in new consumer > -- > > Key: KAFKA-2515 > URL: https://issues.apache.org/jira/browse/KAFKA-2515 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Jun Rao >Assignee: Onur Karaman > Fix For: 0.9.0.0 > > > When there is an oversized message in the broker, it seems that the new > consumer just silently gets stuck. We should at least log an error when this > happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
