[jira] [Commented] (KAFKA-345) Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213469#comment-14213469 ] Jiangjie Qin commented on KAFKA-345: Updated reviewboard https://reviews.apache.org/r/28025/diff/ against branch origin/trunk Add a listener to ZookeeperConsumerConnector to get notified on rebalance events Key: KAFKA-345 URL: https://issues.apache.org/jira/browse/KAFKA-345 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.7, 0.8.0 Reporter: Peter Romianowski Attachments: KAFKA-345.patch, KAFKA-345.patch, KAFKA-345_2014-11-15_01:00:55.patch A sample use-case In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28025: Patch for KAFKA-345
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28025/ --- (Updated Nov. 15, 2014, 9:01 a.m.) Review request for kafka. Bugs: KAFKA-345 https://issues.apache.org/jira/browse/KAFKA-345 Repository: kafka Description (updated) --- Added new unit test. Incorporated Joel's comments Diffs (updated) - core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala f476973eeff653473a60c3ecf36e870e386536bc core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java PRE-CREATION core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 1f98db5d692adc113189ec8c75a4fad29d6b6ffe core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala e1d87112a2a587aa3a2f5875f278b276c32f45ac Diff: https://reviews.apache.org/r/28025/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Updated] (KAFKA-345) Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-345: --- Attachment: KAFKA-345_2014-11-15_01:00:55.patch Add a listener to ZookeeperConsumerConnector to get notified on rebalance events Key: KAFKA-345 URL: https://issues.apache.org/jira/browse/KAFKA-345 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.7, 0.8.0 Reporter: Peter Romianowski Attachments: KAFKA-345.patch, KAFKA-345.patch, KAFKA-345_2014-11-15_01:00:55.patch A sample use-case In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28025: Patch for KAFKA-345
On Nov. 14, 2014, 11:36 p.m., Joel Koshy wrote: core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java, line 24 https://reviews.apache.org/r/28025/diff/1/?file=763200#file763200line24 Can we instead just have an interface for this? We don't need to provide any default implementation since by default the rebalance listener can be null. I was thinking about this. The reason I finally used class is that the user can just override the method they want without implementing the other methods. But an interface is probably more clean. On Nov. 14, 2014, 11:36 p.m., Joel Koshy wrote: core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java, line 29 https://reviews.apache.org/r/28025/diff/1/?file=763200#file763200line29 It would be worth elaborating the use-case for these hooks (i.e., in the context of mirror maker data loss and how these will be used). Do we need all of these hooks? I checked the new consumer. The rebalance callback has 2 hooks. onPartitionRevoked and onPartitionAssigned. I think we can keep them just for future compatibility. It seems people potentially needs before and after hook (that's why this ticket is initially created). For mirror maker we need the hook after fetchers stopped and before the partition ownerships are released. So it seems we still need 5 hooks... On Nov. 14, 2014, 11:36 p.m., Joel Koshy wrote: core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java, line 48 https://reviews.apache.org/r/28025/diff/1/?file=763200#file763200line48 It seems this would be more useful if we pass in the assignment itself. Also, afterPartitionAssignment would be a better name. Yes, I realized this... Actually I will need the assignment for later on design in KET for topic deletion. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28025/#review61556 --- On Nov. 15, 2014, 9:01 a.m., Jiangjie Qin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28025/ --- (Updated Nov. 15, 2014, 9:01 a.m.) Review request for kafka. Bugs: KAFKA-345 https://issues.apache.org/jira/browse/KAFKA-345 Repository: kafka Description --- Added new unit test. Incorporated Joel's comments Diffs - core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala f476973eeff653473a60c3ecf36e870e386536bc core/src/main/scala/kafka/javaapi/consumer/ConsumerRebalanceListener.java PRE-CREATION core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 1f98db5d692adc113189ec8c75a4fad29d6b6ffe core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala e1d87112a2a587aa3a2f5875f278b276c32f45ac Diff: https://reviews.apache.org/r/28025/diff/ Testing --- Thanks, Jiangjie Qin
[jira] [Commented] (KAFKA-345) Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213474#comment-14213474 ] Jiangjie Qin commented on KAFKA-345: Updated reviewboard https://reviews.apache.org/r/28025/diff/ against branch origin/trunk Add a listener to ZookeeperConsumerConnector to get notified on rebalance events Key: KAFKA-345 URL: https://issues.apache.org/jira/browse/KAFKA-345 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.7, 0.8.0 Reporter: Peter Romianowski Attachments: KAFKA-345.patch, KAFKA-345.patch, KAFKA-345_2014-11-15_01:00:55.patch, KAFKA-345_2014-11-15_01:19:56.patch A sample use-case In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-345) Add a listener to ZookeeperConsumerConnector to get notified on rebalance events
[ https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-345: --- Attachment: KAFKA-345_2014-11-15_01:19:56.patch Add a listener to ZookeeperConsumerConnector to get notified on rebalance events Key: KAFKA-345 URL: https://issues.apache.org/jira/browse/KAFKA-345 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.7, 0.8.0 Reporter: Peter Romianowski Attachments: KAFKA-345.patch, KAFKA-345.patch, KAFKA-345_2014-11-15_01:00:55.patch, KAFKA-345_2014-11-15_01:19:56.patch A sample use-case In our scenario we partition events by userid and then apply these to some kind of state machine, that modifies the actual state of a user. So events trigger state transitions. In order to avoid the need of loading user's state upon each event processed, we cache that. But if a user's partition is moved to another consumer and then back to the previous consumer we have stale caches and hell breaks loose. I guess the same kind of problem occurs in other scenarios like counting numbers by user, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1751) handle broker not exists and topic not exists scenarios
[ https://issues.apache.org/jira/browse/KAFKA-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213517#comment-14213517 ] Dmitry Pekar commented on KAFKA-1751: - The implementation of KAFKA-1751 adds validation to check following preconditions for every reassignment operation: - each specified broker should exist; - each specified partition should exist; handle broker not exists and topic not exists scenarios --- Key: KAFKA-1751 URL: https://issues.apache.org/jira/browse/KAFKA-1751 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1751.patch, kafka-1751.patch merged with 1750 to pass by single code review process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1515) Wake-up Sender upon blocked on fetching leader metadata
[ https://issues.apache.org/jira/browse/KAFKA-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1515: - Fix Version/s: (was: 0.9.0) 0.8.2 P Wake-up Sender upon blocked on fetching leader metadata --- Key: KAFKA-1515 URL: https://issues.apache.org/jira/browse/KAFKA-1515 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1515.patch, KAFKA-1515_2014-07-03_10:19:28.patch, KAFKA-1515_2014-07-03_16:43:05.patch, KAFKA-1515_2014-07-07_10:55:58.patch, KAFKA-1515_2014-07-08_11:35:59.patch Currently the new KafkaProducer will not wake up the sender thread upon forcing metadata fetch, and hence if the sender is polling with a long timeout (e.g. the metadata.age period) this wait will usually timeout and fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (KAFKA-1515) Wake-up Sender upon blocked on fetching leader metadata
[ https://issues.apache.org/jira/browse/KAFKA-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1515: - Comment: was deleted (was: P) Wake-up Sender upon blocked on fetching leader metadata --- Key: KAFKA-1515 URL: https://issues.apache.org/jira/browse/KAFKA-1515 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1515.patch, KAFKA-1515_2014-07-03_10:19:28.patch, KAFKA-1515_2014-07-03_16:43:05.patch, KAFKA-1515_2014-07-07_10:55:58.patch, KAFKA-1515_2014-07-08_11:35:59.patch Currently the new KafkaProducer will not wake up the sender thread upon forcing metadata fetch, and hence if the sender is polling with a long timeout (e.g. the metadata.age period) this wait will usually timeout and fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 27990: Patch for KAFKA-1751
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27990/#review61644 --- Looks great! Few comments and nitpicks below. core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103376 Should be unnecessary core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103378 _1 is topicAndPartition and _2 is list of brokers we are going to assign them to? I'd use matching to give them explicit names. Our code guide discourages _1 and _2 unless its really obvious. core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103379 I think we can do better here - more readable and better use of Scala's lists. We are just looking for items in topicsToReassign that don't exist in topicPartitionsToReassign, right? Why not convert both to list of topics and then use listA -- listB to find the diff? core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103381 Shouldn't we be validating partitions here too? core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103382 I'd probably optimize it to be a list operation - i.e. take a list of brokers and compare with result of getAllBrokersInCluster - to avoid too many ZK calls. But this isn't exactly a place where optimizations matter, so feel free to leave this alone :) core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/27990/#comment103383 This pattern appears twice (here and in validate), maybe give it its own function. core/src/test/scala/unit/kafka/admin/AdminTest.scala https://reviews.apache.org/r/27990/#comment103388 I really like the test cleanup. Looks much better now. We still need to make sure this test will fail if the reassignment command magically succeeds and not exception is thrown. core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala https://reviews.apache.org/r/27990/#comment103389 Any reason for this change? - Gwen Shapira On Nov. 13, 2014, 2:39 p.m., Dmitry Pekar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27990/ --- (Updated Nov. 13, 2014, 2:39 p.m.) Review request for kafka. Bugs: KAFKA-1751 https://issues.apache.org/jira/browse/KAFKA-1751 Repository: kafka Description --- KAFKA-1751: handle broker not exists and topic not exists scenarios Diffs - core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 979992b68af3723cd229845faff81c641123bb88 core/src/test/scala/unit/kafka/admin/AdminTest.scala e28979827110dfbbb92fe5b152e7f1cc973de400 core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 29cc01bcef9cacd8dec1f5d662644fc6fe4994bc Diff: https://reviews.apache.org/r/27990/diff/ Testing --- Thanks, Dmitry Pekar
[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerOffsetChecker
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-313: - Summary: Add JSON/CSV output and looping options to ConsumerOffsetChecker (was: Add JSON output and looping options to ConsumerOffsetChecker) Add JSON/CSV output and looping options to ConsumerOffsetChecker Key: KAFKA-313 URL: https://issues.apache.org/jira/browse/KAFKA-313 Project: Kafka Issue Type: Improvement Reporter: Dave DeMaagd Assignee: Ashish Kumar Singh Priority: Minor Labels: newbie, patch Fix For: 0.8.3 Attachments: KAFKA-313-2012032200.diff Adds: * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately) * '--asjson' - display as a JSON string instead of the more human readable output format. Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached. Impacted files: core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 28096: KAFKA-313: Add JSON/CSV output and looping options to ConsumerOffsetChecker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28096/ --- Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy. Bugs: KAFKA-313 https://issues.apache.org/jira/browse/KAFKA-313 Repository: kafka Description --- KAFKA-313: Add JSON/CSV output and looping options to ConsumerOffsetChecker Diffs - core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala d1e7c434e77859d746b8dc68dd5d5a3740425e79 Diff: https://reviews.apache.org/r/28096/diff/ Testing --- Ran ConsumerOffsetChecker with different combinations of --output.format and --loop options. Thanks, Ashish Singh
[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerOffsetChecker
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-313: - Attachment: KAFKA-313.patch Adds --loop N and --output.format options. Supports csv and json output formats. Add JSON/CSV output and looping options to ConsumerOffsetChecker Key: KAFKA-313 URL: https://issues.apache.org/jira/browse/KAFKA-313 Project: Kafka Issue Type: Improvement Reporter: Dave DeMaagd Assignee: Ashish Kumar Singh Priority: Minor Labels: newbie, patch Fix For: 0.8.3 Attachments: KAFKA-313-2012032200.diff, KAFKA-313.patch Adds: * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately) * '--asjson' - display as a JSON string instead of the more human readable output format. Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached. Impacted files: core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot
[
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213707#comment-14213707
]
Ashish Kumar Singh commented on KAFKA-1664:
---
[~gwenshap] I am thinking of taking a look at this. If it is OK with you,
kindly assign the JIRA to me.
Kafka does not properly parse multiple ZK nodes with non-root chroot
Key: KAFKA-1664
URL: https://issues.apache.org/jira/browse/KAFKA-1664
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Ricky Saltzer
Assignee: Gwen Shapira
Priority: Minor
Labels: newbie
When using a non-root ZK directory for Kafka, if you specify multiple ZK
servers, Kafka does not seem to properly parse the connection string.
*Error*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
--topic test-topic
[2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:
(kafka.consumer.ConsoleConsumer$)
java.lang.IllegalArgumentException: Path length must be 0
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.init(ZookeeperConsumerConnector.scala:755)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}
*Working*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KAFKA-1773) Add a tool to check available consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh reassigned KAFKA-1773: - Assignee: Ashish Kumar Singh Add a tool to check available consumer groups - Key: KAFKA-1773 URL: https://issues.apache.org/jira/browse/KAFKA-1773 Project: Kafka Issue Type: Improvement Components: core Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Right now ConsumerOffsetChecker expects consumer group. However, there is no tool to get available consumer groups. This JIRA intends to add a tool to get available consumer groups from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerOffsetChecker
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213746#comment-14213746 ] Ashish Kumar Singh commented on KAFKA-313: -- [~jjkoshy] while working on this I realized that user needs to know consumer groups available at a given time. I could not find an easy way to do this. A tool to get this information will be handy, created KAFKA-1773. Add JSON/CSV output and looping options to ConsumerOffsetChecker Key: KAFKA-313 URL: https://issues.apache.org/jira/browse/KAFKA-313 Project: Kafka Issue Type: Improvement Reporter: Dave DeMaagd Assignee: Ashish Kumar Singh Priority: Minor Labels: newbie, patch Fix For: 0.8.3 Attachments: KAFKA-313-2012032200.diff, KAFKA-313.patch Adds: * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately) * '--asjson' - display as a JSON string instead of the more human readable output format. Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached. Impacted files: core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 28100: KAFKA-1773: Add a tool to check available consumer groups
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28100/ --- Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy. Bugs: KAFKA-1773 https://issues.apache.org/jira/browse/KAFKA-1773 Repository: kafka Description --- KAFKA-1773: Add a tool to check available consumer groups Diffs - core/src/main/scala/kafka/tools/ConsumerGroupChecker.scala PRE-CREATION Diff: https://reviews.apache.org/r/28100/diff/ Testing --- Tested with console producer-consumer and one ZK instance. Thanks, Ashish Singh
[jira] [Comment Edited] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerOffsetChecker
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213706#comment-14213706 ] Ashish Kumar Singh edited comment on KAFKA-313 at 11/15/14 9:00 PM: RB: https://reviews.apache.org/r/28096/ Adds --loop N and --output.format options. Supports csv and json output formats. was (Author: singhashish): Adds --loop N and --output.format options. Supports csv and json output formats. Add JSON/CSV output and looping options to ConsumerOffsetChecker Key: KAFKA-313 URL: https://issues.apache.org/jira/browse/KAFKA-313 Project: Kafka Issue Type: Improvement Reporter: Dave DeMaagd Assignee: Ashish Kumar Singh Priority: Minor Labels: newbie, patch Fix For: 0.8.3 Attachments: KAFKA-313-2012032200.diff, KAFKA-313.patch Adds: * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately) * '--asjson' - display as a JSON string instead of the more human readable output format. Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached. Impacted files: core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1773) Add a tool to check available consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-1773: -- Status: Patch Available (was: Open) Add a tool to check available consumer groups - Key: KAFKA-1773 URL: https://issues.apache.org/jira/browse/KAFKA-1773 Project: Kafka Issue Type: Improvement Components: core Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Right now ConsumerOffsetChecker expects consumer group. However, there is no tool to get available consumer groups. This JIRA intends to add a tool to get available consumer groups from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1773) Add a tool to check available consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213760#comment-14213760 ] Ashish Kumar Singh commented on KAFKA-1773: --- RB: https://reviews.apache.org/r/28100/ Add a tool to check available consumer groups - Key: KAFKA-1773 URL: https://issues.apache.org/jira/browse/KAFKA-1773 Project: Kafka Issue Type: Improvement Components: core Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Right now ConsumerOffsetChecker expects consumer group. However, there is no tool to get available consumer groups. This JIRA intends to add a tool to get available consumer groups from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerOffsetChecker
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-313: - Status: Patch Available (was: Open) Add JSON/CSV output and looping options to ConsumerOffsetChecker Key: KAFKA-313 URL: https://issues.apache.org/jira/browse/KAFKA-313 Project: Kafka Issue Type: Improvement Reporter: Dave DeMaagd Assignee: Ashish Kumar Singh Priority: Minor Labels: newbie, patch Fix For: 0.8.3 Attachments: KAFKA-313-2012032200.diff, KAFKA-313.patch Adds: * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately) * '--asjson' - display as a JSON string instead of the more human readable output format. Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached. Impacted files: core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28100: Patch for KAFKA-1773
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28100/ --- (Updated Nov. 16, 2014, 1:48 a.m.) Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy. Summary (updated) - Patch for KAFKA-1773 Bugs: KAFKA-1773 https://issues.apache.org/jira/browse/KAFKA-1773 Repository: kafka Description --- KAFKA-1773: Add a tool to check available consumer groups Diffs (updated) - core/src/main/scala/kafka/tools/ConsumerGroupChecker.scala PRE-CREATION Diff: https://reviews.apache.org/r/28100/diff/ Testing --- Tested with console producer-consumer and one ZK instance. Thanks, Ashish Singh
[jira] [Updated] (KAFKA-1773) Add a tool to check available consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-1773: -- Attachment: KAFKA-1773.patch Add a tool to check available consumer groups - Key: KAFKA-1773 URL: https://issues.apache.org/jira/browse/KAFKA-1773 Project: Kafka Issue Type: Improvement Components: core Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Attachments: KAFKA-1773.patch Right now ConsumerOffsetChecker expects consumer group. However, there is no tool to get available consumer groups. This JIRA intends to add a tool to get available consumer groups from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP
Joe Stein created KAFKA-1774: Summary: REPL and Shell Client for Admin Message RQ/RP Key: KAFKA-1774 URL: https://issues.apache.org/jira/browse/KAFKA-1774 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein We should have a REPL we can work in and execute the commands with the arguments. With this we can do: ./kafka.sh --shell kafkaattach cluster -b localhost:9092; kafkadescribe topic sampleTopicNameForExample; the command line version can work like it does now so folks don't have to re-write all of their tooling. kafka.sh --topics --everything the same like kafka-topics.sh is kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh is I think this part should reside in the client folder and be written in Go or Python (or Both) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1775) Re-factor TopicCommand into thew handerAdminMessage call
Joe Stein created KAFKA-1775: Summary: Re-factor TopicCommand into thew handerAdminMessage call Key: KAFKA-1775 URL: https://issues.apache.org/jira/browse/KAFKA-1775 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein kafka-topic.sh should become kafka --topic --verything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for [1] https://issues.apache.org/jira/browse/KAFKA-1772 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1775) Re-factor TopicCommand into thew handerAdminMessage call
[ https://issues.apache.org/jira/browse/KAFKA-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1775: - Description: kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for [1] https://issues.apache.org/jira/browse/KAFKA-1772 was: kafka-topic.sh should become kafka --topic --verything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for [1] https://issues.apache.org/jira/browse/KAFKA-1772 Re-factor TopicCommand into thew handerAdminMessage call - Key: KAFKA-1775 URL: https://issues.apache.org/jira/browse/KAFKA-1775 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for [1] https://issues.apache.org/jira/browse/KAFKA-1772 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1776) re-factor out existing tools that have been implemented behind the CLI
Joe Stein created KAFKA-1776: Summary: re-factor out existing tools that have been implemented behind the CLI Key: KAFKA-1776 URL: https://issues.apache.org/jira/browse/KAFKA-1776 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Priority: Minor Fix For: 0.9.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1775) Re-factor TopicCommand into thew handerAdminMessage call
[ https://issues.apache.org/jira/browse/KAFKA-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1775: - Description: kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for I think we should not remove the existing kafka-topic.sh and preserve the existing functionality (with as little code duplication as possible) until 0.9 (and there we can remove it after folks have used it for a release or two and feedback and the rest)[2] [1] https://issues.apache.org/jira/browse/KAFKA-1772 [2] https://issues.apache.org/jira/browse/KAFKA-1776 was: kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for I think we should not remove the existing kafka-topic.sh and preserve the existing functionality (with as little code duplication as possible) until 0.9 (and there we can remove it after folks have used it for a release or two and feedback and the rest)[2] [1] https://issues.apache.org/jira/browse/KAFKA-1772 [2] https://issues.apache.org/jira/browse/KAFKA-1772 Re-factor TopicCommand into thew handerAdminMessage call - Key: KAFKA-1775 URL: https://issues.apache.org/jira/browse/KAFKA-1775 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for I think we should not remove the existing kafka-topic.sh and preserve the existing functionality (with as little code duplication as possible) until 0.9 (and there we can remove it after folks have used it for a release or two and feedback and the rest)[2] [1] https://issues.apache.org/jira/browse/KAFKA-1772 [2] https://issues.apache.org/jira/browse/KAFKA-1776 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1775) Re-factor TopicCommand into thew handerAdminMessage call
[ https://issues.apache.org/jira/browse/KAFKA-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1775: - Description: kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for I think we should not remove the existing kafka-topic.sh and preserve the existing functionality (with as little code duplication as possible) until 0.9 (and there we can remove it after folks have used it for a release or two and feedback and the rest)[2] [1] https://issues.apache.org/jira/browse/KAFKA-1772 [2] https://issues.apache.org/jira/browse/KAFKA-1772 was: kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for [1] https://issues.apache.org/jira/browse/KAFKA-1772 Re-factor TopicCommand into thew handerAdminMessage call - Key: KAFKA-1775 URL: https://issues.apache.org/jira/browse/KAFKA-1775 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 kafka-topic.sh should become kafka --topic --everything else the same from the CLI perspective so we need to have the calls from the byte lalery get fed into that same code (few changes as possible called from the handleAdmin call after deducing what Utility[1] it is operating for I think we should not remove the existing kafka-topic.sh and preserve the existing functionality (with as little code duplication as possible) until 0.9 (and there we can remove it after folks have used it for a release or two and feedback and the rest)[2] [1] https://issues.apache.org/jira/browse/KAFKA-1772 [2] https://issues.apache.org/jira/browse/KAFKA-1772 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1777) Re-factor reasign-partitions into CLI
Joe Stein created KAFKA-1777: Summary: Re-factor reasign-partitions into CLI Key: KAFKA-1777 URL: https://issues.apache.org/jira/browse/KAFKA-1777 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1778) Create new re-elect controller admin function
Joe Stein created KAFKA-1778: Summary: Create new re-elect controller admin function Key: KAFKA-1778 URL: https://issues.apache.org/jira/browse/KAFKA-1778 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 kafka --controller --elect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 27684: Patch for KAFKA-1743
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27684/ --- (Updated Nov. 16, 2014, 6:42 a.m.) Review request for kafka. Bugs: KAFKA-1743 https://issues.apache.org/jira/browse/KAFKA-1743 Repository: kafka Description (updated) --- def commitOffsets method added to make ConsumerConnector backward compatible; Addressing Jun's comments Diffs (updated) - core/src/main/scala/kafka/consumer/ConsumerConnector.scala 07677c1c26768ef9c9032626180d0015f12cb0e0 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala f476973eeff653473a60c3ecf36e870e386536bc core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 1f98db5d692adc113189ec8c75a4fad29d6b6ffe Diff: https://reviews.apache.org/r/27684/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Updated] (KAFKA-1743) ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
[ https://issues.apache.org/jira/browse/KAFKA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1743: --- Attachment: KAFKA-1743_2014-11-16_12:11:51.patch ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible --- Key: KAFKA-1743 URL: https://issues.apache.org/jira/browse/KAFKA-1743 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1743.patch, KAFKA-1743_2014-11-08_11:49:31.patch, KAFKA-1743_2014-11-14_22:29:21.patch, KAFKA-1743_2014-11-16_12:11:51.patch In 0.8.1.x, ConsumerConnector has the following api: def commitOffsets This is changed to the following in 0.8.2 and breaks compatibility def commitOffsets(retryOnFailure: Boolean = true) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1743) ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible
[ https://issues.apache.org/jira/browse/KAFKA-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213869#comment-14213869 ] Manikumar Reddy commented on KAFKA-1743: Updated reviewboard https://reviews.apache.org/r/27684/diff/ against branch origin/0.8.2 ConsumerConnector.commitOffsets in 0.8.2 is not backward compatible --- Key: KAFKA-1743 URL: https://issues.apache.org/jira/browse/KAFKA-1743 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1743.patch, KAFKA-1743_2014-11-08_11:49:31.patch, KAFKA-1743_2014-11-14_22:29:21.patch, KAFKA-1743_2014-11-16_12:11:51.patch In 0.8.1.x, ConsumerConnector has the following api: def commitOffsets This is changed to the following in 0.8.2 and breaks compatibility def commitOffsets(retryOnFailure: Boolean = true) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 27684: Patch for KAFKA-1743
On Nov. 14, 2014, 7:29 p.m., Jun Rao wrote: Thanks for the patch. For clarity, in ZookeeperConsumerConnector, instead of having the following, def commitOffsets(isAutoCommit: Boolean = true) could we break it into two separate methods, same as what's defined in ConsumerConnector? Done. Added the following APIs in ZookeeperConsumerConnector. def commitOffsets(retryOnFailure: Boolean) def commitOffsets - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27684/#review61486 --- On Nov. 16, 2014, 6:42 a.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27684/ --- (Updated Nov. 16, 2014, 6:42 a.m.) Review request for kafka. Bugs: KAFKA-1743 https://issues.apache.org/jira/browse/KAFKA-1743 Repository: kafka Description --- def commitOffsets method added to make ConsumerConnector backward compatible; Addressing Jun's comments Diffs - core/src/main/scala/kafka/consumer/ConsumerConnector.scala 07677c1c26768ef9c9032626180d0015f12cb0e0 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala f476973eeff653473a60c3ecf36e870e386536bc core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 1f98db5d692adc113189ec8c75a4fad29d6b6ffe Diff: https://reviews.apache.org/r/27684/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Updated] (KAFKA-1776) Re-factor out existing tools that have been implemented behind the CLI
[ https://issues.apache.org/jira/browse/KAFKA-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1776: - Summary: Re-factor out existing tools that have been implemented behind the CLI (was: re-factor out existing tools that have been implemented behind the CLI) Re-factor out existing tools that have been implemented behind the CLI -- Key: KAFKA-1776 URL: https://issues.apache.org/jira/browse/KAFKA-1776 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Priority: Minor Fix For: 0.9.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
