[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984933#comment-14984933 ] Andrii Biletskyi commented on KAFKA-1694: - [~granthenke], it was decided to split the work into 3 phases: api, admin client, cli. The Phase 1 was implemented under KAFKA-2229, the patch was moved to github (https://github.com/apache/kafka/pull/223). There were some minor comments under this pull request, they were fixed, though not rebased. IMO it lacks some deeper review and maybe testing. In short, you can pickup this ticket. I'm happy to help to close this issue asap too. If anything is needed (i.e. rebase) let me know. > kafka command line and centralized operations > - > > Key: KAFKA-1694 > URL: https://issues.apache.org/jira/browse/KAFKA-1694 > Project: Kafka > Issue Type: Bug >Reporter: Joe Stein >Assignee: Andrii Biletskyi >Priority: Critical > Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, > KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, > KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, > KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, > KAFKA-1772_1802_1775_1774_v2.patch > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983366#comment-14983366 ] Grant Henke commented on KAFKA-1694: [~abiletskyi], are you still able to work on this? I would like to pick the work up if not. I am looking forward to getting this functionality into Kafka as soon as possible. > kafka command line and centralized operations > - > > Key: KAFKA-1694 > URL: https://issues.apache.org/jira/browse/KAFKA-1694 > Project: Kafka > Issue Type: Bug >Reporter: Joe Stein >Assignee: Andrii Biletskyi >Priority: Critical > Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, > KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, > KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, > KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, > KAFKA-1772_1802_1775_1774_v2.patch > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359409#comment-14359409 ] Andrii Biletskyi commented on KAFKA-1694: - Hi, The list of changes from the latest patch: - removed MaybeOf type and changed RQ/RP protocol - switched over to java protocol definitions (removed scala classes) - removed workaround for KAFKA-1867 (since it was fixed and committed) - code review fixes What's left (pending discussion on mailing list): - Batching Admin Operations - Remove Cluster Metadata, evolve TopicMetadata to V1 instead - Introduce fine-grained error codes instead of single AdminRequestFailedError I also update KIP-4 accordingly and added proposed changes to cover all pending items. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358489#comment-14358489 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327691#comment-14327691 ] Guozhang Wang commented on KAFKA-1694: -- Sounds great, thanks! kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326171#comment-14326171 ] Andrii Biletskyi commented on KAFKA-1694: - [~guozhang], yes, initially I did split it to several patches. The problem is that almost all tickets depend on each other. Also I believe that single patch lets you better understand the whole picture (and the use case - CLI) so people can comment, argue in mail list. But I totally agree with you this way we can miss some hidden issues. I would prefer to collect some feedback to be sure we more or less agree on approach and after that I can split this big feature to sub-patches. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326172#comment-14326172 ] Andrii Biletskyi commented on KAFKA-1694: - [~guozhang], yes, initially I did split it to several patches. The problem is that almost all tickets depend on each other. Also I believe that single patch lets you better understand the whole picture (and the use case - CLI) so people can comment, argue in mail list. But I totally agree with you this way we can miss some hidden issues. I would prefer to collect some feedback to be sure we more or less agree on approach and after that I can split this big feature to sub-patches. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324755#comment-14324755 ] Guozhang Wang commented on KAFKA-1694: -- [~abiletskyi], I am wondering if we can split the current RBs into multiple ones following the subtasks structure. Concerns with having one big RB is that it is hard to review, and also very hard to commit: as there will likely have some hidden issues with such big changes, and upon finding them after we commit, if it is not easily fix-forwardable we have to revert the whole thing. What do you think? kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277126#comment-14277126 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276910#comment-14276910 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14275577#comment-14275577 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273588#comment-14273588 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273773#comment-14273773 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258468#comment-14258468 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255566#comment-14255566 ] Andrii Biletskyi commented on KAFKA-1694: - Created reviewboard https://reviews.apache.org/r/29301/diff/ against branch origin/trunk kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245282#comment-14245282 ] Andrii Biletskyi commented on KAFKA-1694: - [~junrao]: Thank you for you comments. I agree with all you remarks, we also think about breaking AdminRequest into separate requests as a preferable solution. I created and uploaded this patch to see whether we go at least right direction in building shell for admin requests, as I understand there are no major concerns for now about the structure/approach so I'll submit this all with patch review tool and we can follow our usual procedure. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245091#comment-14245091 ] Jun Rao commented on KAFKA-1694: Thanks for the patch. A few comments. 10. I still don't think having a single admin request/response is a good idea. Since the args for different sub admin requests are quite different, it's very hard to reason about what the format in args is. Ideally, we should be able to figure out the request format just from the protocol definition. It seems to me it's cleaner to just create the following standalone requests/responses. 1. create topic 2. alter topic 3. delete topic 4. list topic 5. describe topic (this will be used to replace TopicMetadataRequest eventually) 6. describe cluster (return all brokers and the controller) 11. We can reuse the java request objects in the scala request (see HeartbeatRequestAndHeader.scala as an example). 12. describe topic : I am not sure that we need to include the following options in the request. We can just return the info (replicas, isrs, etc) of all partitions and let the client decide what to do with them. reportUnderReplicatedPartitions reportUnavailablePartitions Since this is a large patch, could you submit an RB using the patch review tool? kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
