[jira] [Commented] (KAFKA-5327) Console Consumer should only poll for up to max messages
[ https://issues.apache.org/jira/browse/KAFKA-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035261#comment-16035261 ] ASF GitHub Bot commented on KAFKA-5327: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3148 > Console Consumer should only poll for up to max messages > > > Key: KAFKA-5327 > URL: https://issues.apache.org/jira/browse/KAFKA-5327 > Project: Kafka > Issue Type: Improvement > Components: tools >Reporter: Dustin Cote >Assignee: huxi >Priority: Minor > Fix For: 0.11.0.0 > > > The ConsoleConsumer has a --max-messages flag that can be used to limit the > number of messages consumed. However, the number of records actually consumed > is governed by max.poll.records. This means you see one message on the > console, but your offset has moved forward a default of 500, which is kind of > counterintuitive. It would be good to only commit offsets for messages we > have printed to the console. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5327) Console Consumer should only poll for up to max messages
[ https://issues.apache.org/jira/browse/KAFKA-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032756#comment-16032756 ] ASF GitHub Bot commented on KAFKA-5327: --- GitHub user amethystic reopened a pull request: https://github.com/apache/kafka/pull/3148 KAFKA-5327: ConsoleConsumer should manually commit offsets... KAFKA-5327: ConsoleConsumer should manually commit offsets for those records it really consumed. Currently it leaves this job to the automatic offset commit scheme where some unread messages will be passed if `--max-messages` is set. You can merge this pull request into a Git repository by running: $ git pull https://github.com/amethystic/kafka KAFKA-5327_ConsoleConsumer_distable_autocommit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3148.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3148 commit fc884eb0bd9940c3914df5e74eae25ff12bb14b9 Author: amethystic Date: 2017-05-26T03:12:17Z KAFKA-5327: ConsoleConsumer explicitly set `max.poll.records` if `--max-messages` is set. commit 142c6a7cbe5a769d1c582ddc70f0b03269c95b1e Author: amethystic Date: 2017-05-26T07:06:37Z KAFKA-5327: ConsoleConsumer's new consumer commits offsets for those consumed messages commit 691cc1dd9080e677f2207c316bd7daef889eb918 Author: amethystic Date: 2017-05-31T01:04:46Z merged trunk code commit 04a72cd9ca847e58cad49873a149ad9e915b7246 Author: amethystic Date: 2017-05-31T01:59:54Z Merge branch 'trunk' into KAFKA-5327_ConsoleConsumer_distable_autocommit commit 13ada359e7bd6028a976e6771ed5042f33c269c1 Author: amethystic Date: 2017-06-01T10:10:12Z Cache partitions to which messages are sent in case of creating too many unnecessary objects to burden the young GC commit d21b3762b282b8272f059e89875d5f023c3e199d Author: amethystic Date: 2017-06-01T10:12:03Z Merge remote-tracking branch 'origin/KAFKA-5327_ConsoleConsumer_distable_autocommit' into KAFKA-5327_ConsoleConsumer_distable_autocommit commit 7f396c842e0ee70c780ad505e81bc77fe894c780 Author: amethystic Date: 2017-06-01T10:15:19Z Merge remote-tracking branch 'upstream/trunk' into KAFKA-5327_ConsoleConsumer_distable_autocommit commit 4a4ecc8326450a77569f5049093df02a8bd55a4b Author: amethystic Date: 2017-06-01T10:25:56Z KAFKA-5327: ConsoleConsumer should manually commit offsets for those really consumed messages. > Console Consumer should only poll for up to max messages > > > Key: KAFKA-5327 > URL: https://issues.apache.org/jira/browse/KAFKA-5327 > Project: Kafka > Issue Type: Improvement > Components: tools >Reporter: Dustin Cote >Assignee: huxi >Priority: Minor > > The ConsoleConsumer has a --max-messages flag that can be used to limit the > number of messages consumed. However, the number of records actually consumed > is governed by max.poll.records. This means you see one message on the > console, but your offset has moved forward a default of 500, which is kind of > counterintuitive. It would be good to only commit offsets for messages we > have printed to the console. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5327) Console Consumer should only poll for up to max messages
[ https://issues.apache.org/jira/browse/KAFKA-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032742#comment-16032742 ] ASF GitHub Bot commented on KAFKA-5327: --- Github user amethystic closed the pull request at: https://github.com/apache/kafka/pull/3148 > Console Consumer should only poll for up to max messages > > > Key: KAFKA-5327 > URL: https://issues.apache.org/jira/browse/KAFKA-5327 > Project: Kafka > Issue Type: Improvement > Components: tools >Reporter: Dustin Cote >Assignee: huxi >Priority: Minor > > The ConsoleConsumer has a --max-messages flag that can be used to limit the > number of messages consumed. However, the number of records actually consumed > is governed by max.poll.records. This means you see one message on the > console, but your offset has moved forward a default of 500, which is kind of > counterintuitive. It would be good to only commit offsets for messages we > have printed to the console. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5327) Console Consumer should only poll for up to max messages
[ https://issues.apache.org/jira/browse/KAFKA-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025743#comment-16025743 ] ASF GitHub Bot commented on KAFKA-5327: --- GitHub user amethystic opened a pull request: https://github.com/apache/kafka/pull/3148 KAFKA-5327: ConsoleConsumer explicitly set `max.poll.records`... KAFKA-5327: ConsoleConsumer explicitly set `max.poll.records` if `--max-messages` is set. You can merge this pull request into a Git repository by running: $ git pull https://github.com/amethystic/kafka KAFKA-5327_ConsoleConsumer_distable_autocommit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3148.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3148 commit fc884eb0bd9940c3914df5e74eae25ff12bb14b9 Author: amethystic Date: 2017-05-26T03:12:17Z KAFKA-5327: ConsoleConsumer explicitly set `max.poll.records` if `--max-messages` is set. > Console Consumer should only poll for up to max messages > > > Key: KAFKA-5327 > URL: https://issues.apache.org/jira/browse/KAFKA-5327 > Project: Kafka > Issue Type: Improvement > Components: tools >Reporter: Dustin Cote >Assignee: huxi >Priority: Minor > > The ConsoleConsumer has a --max-messages flag that can be used to limit the > number of messages consumed. However, the number of records actually consumed > is governed by max.poll.records. This means you see one message on the > console, but your offset has moved forward a default of 500, which is kind of > counterintuitive. It would be good to only commit offsets for messages we > have printed to the console. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
